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FOREWORD 

a.  This  technical  report,  BDM/ABQ-86-0360-TR,  Is  submitted  by  The 
BOM  Corporation,  1801  Randolph  Road  SE,  Albuquerque,  New  Mexico  87106 
to  the  Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air 
Force  Base,  Albuquerque,  New  Mexico  87117-7001.  This  submission  is 
in  compliance  with  the  requirements  of  paragraph  7.3  of  Subtask 
Statement  412/01,  titled  "Risk  Assessment  Methodology  for  Software 
Supportablllty  (RAMSS):  Pilot  Evaluation  Results  and  Methodology 
Refinement." 


b.  This  report  is  the  result  of  effort  by  Dr.  David  Peercy  (Tech¬ 
nical  Leader),  Mr.  Walter  Huebner,  Jr.  (Task  Leader),  Mr.  Donan 
Estill,  and  Ms.  Kelley  Shaw,  of  The  BDM  Corporation.  The  primary 
Subtask  Statement  Project  Officer  is  Capt.  Eric  H.  Tomlin 
(AF0TEC/LG5T);  the  alternate  Subtask  Statement  Project  Officer  is 
Maj.  Gary  R.  Horlbeck  (AF0TEC/LG5T)/ 
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SECTION  I 
INTRODUCTION 


1.1  PURPOSE  OF  REPORT. 

a.  The  purposes  of  this  report  are  to  document: 

(1)  The  pilot  application  of  the  risk  assessment  methodology 
for  software  supportabil ity  (RAMSS)  to  an  on-going  Air 
Force  Operational  Test  and  Evaluation  Center  (AFOTEC) 
program 

(2)  The  refinements  to  the  risk  assessment  methodology  based 
upon  the  pilot  application 

(3)  Jhe  procedures  which  AFOTEC  will  use  in  the  application 
of  the  RAMSS  to  future  programs. 

b.  It  is  intended  that,  to  the  maximum  extent  possible,  the  pro¬ 
cedures  discussed  in  this  report  will  provide  AFOTEC  with  a  completed 
methodology  which  can  be  validated  and  applied  to  future  software 
evaluations  independent  of  contractor  support. 

1.2  OBJECTIVES  OF  RAMSS. 

1.2.1  Overview  of  Objectives. 

a.  AFOTEC  has  the  responsibi 1 ity  for  conducting  operational  test 
and  evaluation  (OT&E)  of  assets  entering  the  Air  Force  inventory. 
AFOTEC  has  developed  and  implemented  various  software  OT&E  methodol¬ 
ogies.  These  methods  have  matured  and  have  become  the  Air  Force 
standard  for  evaluating  software  supportabi 1 ity.  Each  of  these 
developed  methods  evaluates  specific  characteristics  of  the 
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supportability  aspects  of  delivered  software  and  software  support 
resources.  These  stand-alone  evaluations  provide  AFOTEC  with  infor¬ 
mation  to  Identify  particular  software  supportability  deficiencies, 
but  do  not  identify  overall  risk  associated  with  contractor  or 
military  ownership  and  organic  maintenance  of  contractor-delivered 
software. 

b.  The  development  of  the  RAMSS  has  resulted  from  AFOTEC' s  con¬ 
cern  about  the  need  for  a  risk  assessment  method  which  provides  soft¬ 
ware  testers  with  areas  which  require  testing  emphasis  and  decision 
makers  with  an  assessment  of  the  software  supportability  risk.  The 
objectives  of  the  RAMSS  can  be  classified  as  both  programmatic  and 
technical.  In  particular,  the  programmatic  objectives  are  to  provide 
a  method  which  allows: 

(1)  Early  planning  and  trade-off  studies  for  software  support 
resource  requirements 

(2)  Early  visibility  of  requirements  for  expected  software 
support  actions 

(3)  Early  view  of  potential  software  support  management 
problems 

(4)  Capability  to  trace  software  supportability  risk  measures 
throughout  the  system  life  cycle. 

c.  The  technical  objectives,  which  complement  the  program  objec¬ 
tives,  are  that  the  method  should: 

(1)  Have  a  technical  depth  and  resulting  format  appropriate 
to  adequately  assist  decision  makers 
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(2)  Integrate  at  least  the  current  AFOTEC  evaluation  method¬ 
ologies 

(3)  Have  enough  accuracy  and  repeatability  to  warrant  confi¬ 
dence  in  the  results 

(4)  Be  based  upon  a  sound  theoretical  software  and  risk 

assessment  foundation 

(5)  Allow  for  determination  of  what  acceptable  level  of  risk 
means  depending  upon  the  identity  of  the  risk  agent  and 
the  software  supportability  requirements. 

d.  The  following  subsections  will  give  the  reader  a  brief  back¬ 
ground  review  of  the  RAMSS  development  and  discuss  the  major,  elements 
of  the  method. 

1.2.2  Concept  Development. 

a.  Since  1982,  AFOTEC  has  been  analyzing  the  problem  of  how  to 
assess  the  risk  to  the  Air  Force  of  supporting  software  acquired  for 
weapon  systems.  A  concept  for  computer  resources  risk  assessment 
during  operational  test  and  evaluation  was  proposed  in  1983 

(reference  1.4.11).  Several  issues  evolved  from  this  proposal. 

First,  the  assessed  risk  should  reflect  software  supportability 

impact  upon  the  system  at  a  level  appropriate  for  AFOTEC  reporting 

requirements.  Second,  supportability  is  a  concern  for  both  the  user 
and  the  supporter.  Any  defined  risk  of  software  supportability 

should  reflect  some  aspect  of  user  risk  and  supporter  risk.  Third, 

current  AFOTEC  methods  of  evaluating  software  supportabi 1 ity  should 
be  integrated  into  the  risk  assessment  method.  Also,  the  risk 

assessment  method  should  be  adaptable  to  include  other  AFOTEC 
concerns  such  as  software  maturity  and  software  reliability. 
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b.  Tills  initial  concept  proposal  provided  AFOTEC  with  justifica¬ 
tion  to  study  the  feasibility  of  developing  and  implementing  a  risk 
assessment  methodology  for  software  supportabil ity  (RAMSS).  The 
approach  for  this  study  (references  1.4.2,  1.4.3,  1.4.4)  included: 

(1)  Literature  review  and  assemblage  of  a  data  base  of 
relevant  tools,  techniques  and  methods 

(2)  Analysis  of  relevant  tools,  techniques,  and  methods  for 
feasibility  of  application  to  AFOTEC's  needs 

(3)  Development  of  a  framework  for  assessing  software  sup- 
portability  risk  along  with  a  preliminary  set  of  risk 
measures. 

c.  The  primary  conclusion  from  this  feasibility  study  was  that  a 
RAMSS  could  be  developed  based  upon  the  framework  derived  as  part  of 
the  study.  However,  there  were  still  several  technical  issues  which 
needed  to  be  resolved.  Of  these  issues,  the  major  one  concerned  the 
need  to  establish  a  baseline  against  which  to  measure  risk.  Since 
risk  was  defined  (for  this  study)  as  "the  potential  for  realization 
of  unwanted,  negative  consequences  of  an  event,"  it  was  necessary  to 
have  a  baseline  of  software  support  activities  in  order  to  tell  when 
a  consequence  may  be  negative.  This  baseline,  called  an  historical 
maintenance  profile,  reflects  how  software  support  resources  are 
being  used  to  perform  the  software  support  activities.  Given  this 
information,  the  framework  recommended  by  the  feasibility  study  could 
be  used  to  compute  measures  of  risk  and  incorporate  the  issues 
proposed  in  1983. 

1.2.3  Methodology  Requirements  (Inputs).  Figure  1-1  illustrates 
interfaces  with  the  RAMSS.  The  inputs  consist  of: 
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The  historical  profile  of  software  maintenance  activity 
and  software  supportability  evaluations 

-A»-< 

(2)  A  user/supporter  baseline  estimate  of  planned  software 
maintenance  changes  and  support  resource  requirements  for 
the  software  system  being  evaluated 

(3)  An  evaluation  of  software  support  capabilities  using 

current  AFOTEC  methods. 

1.2.4  Methodology  Analysis.  The  RAMSS  inputs  are  combined  and 

analyzed,  and  measures  of  risk  are  computed  for  the  system  being 
evaluated. 

1.2.5  Methodology  Benefits  (Results). 

a.  The  major,  results  of  the  RAMSS  are  also  illustrated  in 

figure  1-1.  These  results  include: 

(1)  The  software  supportability  risk  measure  which  quantifies 

the  probability  of  the  user/supporter  baseline  estimate 
not  being  accomplished  with  current  software  support 

capabilities 

(2)  The  capability  to  identify  the  impact  of  the  software 
supportability  risk  as  high,  medium,  or  low 

(3)  The  identification  of  the  drivers  of  the  software  sup¬ 
portability  risk 

(4)  The  projection  of  alternative  choices  for  risk  reduction 
(for  instance,  by  improving  certain  aspects  of  current  or 
projected  software  support  capabilities). 
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b.  Mlfghk-thls  Information,  the  decision  maker  can  assess  the 
effect  software  supportability  upon  system  suitability  and 
effectiveness.  In  addition,  detailed  data  are  available  to  help 
answer  specific  questions  such  as  why  particular  areas  of  software 
supportability  are  drivers  and  how  the  software  supportability  risk 
can  be  reduced  to  an  acceptable  level. 

1.2.6  Baseline  Definition  and  Application. 

a.  As  discussed  above,  a  key  element  to  the  risk  assessment  pro¬ 
cess  Is  recognition  that  software  supportability  is  important  both  to 
the  user  and  to  the  supporter  of  the  software.  Therefore  any  risk 
assessment  methodology  which  ignores  the  interests  of  one  of  these 
parties  may  estimate  a  risk  that  is  unacceptable  to  the  other.  In  an 
attempt  to  bridge  this  gap,  the  RAMSS  input  requires  a  User/Supporter 
Baseline  Estimate  be  established,  and  that  the  evaluations  of  soft¬ 
ware  support  capabilities  be  made  against  that  baseline. 

b.  The  User/Supporter  Baseline  Estimate  uses  inputs  from  both  the 
user  (using  command)  and  the  supporter  (supporting  command).  The 
estimate  includes  an  understanding  of  the  software  block  release 
cycle,  projected  software  support  personnel  (numbers  and  types),  and 
anticipated  software  change  request  activity  for  each  block  release. 
Details  of  the  User/Supporter  Baseline  Estimate  are  contained  in 
section  II  of  this  report. 

c.  The  current  AFOTEC  methods  for  evaluating  software  support- 
ability  do  not  consider  in  a  direct  manner  the  effect  of  an  estimated 
baseline.  The  establishment  of  an  estimated  baseline  is  critical  to 
risk  assessment  because  it  (1)  provides  a  means  to  judge  how  well  the 
measured  risk  agrees  with  the  estimated  risk  (which  is,  in  some 
sense,  "acceptable"  to  both  the  user  and  the  supporter),  and 
(2)  quantifies  the  options  required  to  lower  the  measured  and 
acceptable  risks  (a  desired  result  of  the  risk  assessment  process). 
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This  report  documents  what  steps  should  be  taken  during  the  evalua¬ 
tion  of  a  system's  software  supportabi 1 ity  to  ensure  that  the 
User/Supporter  Baseline  Estimate  is  accounted  for  in  the  risk  assess¬ 
ment  process. 

1.3  GENERAL  ORGANIZATION  OF  REPORT. 

The  remainder  of  this  .report  is  organized  into  three  addi¬ 
tional  sections,  plus  a  set  of  appendices  which  provide  useful 
support  information  for  the  RAMSS.  Report  sections  satisfy  the 
following  objectives: 

(1)  Section  II  presents  the  results  of  the  pilot  application 
of  the  RAMSS  to  an  on-going  AFOTEC  program.  The  program 
selected  by  AFOTEC  for  this  application  was  the  JTIDS 
Class  2  Terminal. 

(2)  Section  III  contains  a  discussion  of  the  refinements  to 
the  RAMSS  as  a  result  of  lessons  learned  during  the  pilot 
application.  This  discussion  is  quite  technical  in 
nature,  involving  statistical  analysis  techniques. 

(3)  Section  IV  contains  a  summary  of  the  conclusions  and 
recommendations  of  this  study  and  pilot  application. 

(4)  Appendix  A  contains  a  set  of  briefing  materials  which 
will  be  useful  to  AFOTEC  in  presenting  or  introducing  the 
RAMSS  to  others. 

(5)  Appendix  B  is  an  Evaluator’s  Guide  which  can  serve  as 
stand-alone  reference  material  for  those  who  are  applying 
the  RAMSS  to  future  programs.  This  material  will  guide 
the  evaluator  through  the  necessary  steps  of  RAMSS  and 
discuss  the  integration  of  the  data  into  a  report  which 
concludes  the  entire  process. 
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1.5  TERMS  AND  ABBREVIATIONS. 


AF 

AFB 

AFOTEC 

ALC 

ASSET 

BMOP 

CDRL 

C-E 

COMMANDS 

CPIN 

CRISP 

CRLCMP 

CRMP 


Air  Force 
Air  Force  Base 

Air  Force  Operational  Test  and  Evaluation  Center 

Air  Logistics  Center 

AFOTEC  Software  Support  Evaluation  Tool 

BMDP  Statistical  Software  (NOTE:  BMDP  is  a  name,  not  an 

acronym. ) 

Contract  Data  Requirements  List 
Communications -Electronics 

Communication  and  Navigation  Dynamic  Simulator 
Computer  Program  Identification  Number 
Computer  Resources  Integrated  Support  Plan 
Computer  Resources  Life  Cycle  Management  Plan 
Computer  Resources  Management  Plan 
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CRWG 

Computer  Resources  Working  Group 

CSCI 

Computer  Software  Configuration  Item 

CTP 

Cipher  Text  Processor 

DoO 

Department  of  Defense 

DSE 

Deputy  for  Software  Evaluation 

ECS 

Embedded  Computer  System 

EPROM 

Erasable,  Programmable  Read-Only  Memory 

ESD 

Electronic  Systems  Division 

FCA 

Functional  Configuration  Audit 

FQT 

Formal  Qualification  Test 

HQ-TAC 

Headquarters,  Tactical  Air  Command 

ICPCP 

Indicator  Control  Panel  Control  Program 

IOT1E 

Initial  Operational  Test  and  Evaluation 

ISF 

Integrated  Support  Facility 

MCE 

Modular  Central  Equipment 

NICP 

Network  Interface  Control  Program 

O/S  CMP 

Operational/Support  Configuration  Management  Procedures 

OT&E 

Operational  Test  and  Evaluation 

PCA 

Physical  Configuration  Audit 

PMRT 

Program  Management  Responsibility  Transfer 

PQT 

Preliminary  Qualification  Test 

PTP 

Plain  Text  Processor 

QA 

Quality  Assurance 

QAP 

Questionnaire  Analysis  Program 

RA 

Risk  Assessment 

RAMSS 

Risk  Assessment  Methodology  for  Software  Supportabi 1 ity 

RFP 

Request  For  Proposal 

S/W 

Software 

SICP 

Subscriber  Interface  Control  Program 

SLCP 

Software  Life  Cycle  Process 

SPM 

Software  Product  Maintainability 

ss 

Software  Supportabi 1 ity 

SSR 

Software  Support  Resources 

STM 

Software  Test  Manager 
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TATP  Terminal  Acceptance  Test  Program 

TEMP  Test  and  Evaluation  Master  Plan 

TPO  Test  Plan  Outline 

USBE  User/Supporter  Baseline  Estimate 

WR-ALC  Warner  Robins  Air  Logistics  Center 
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SECTION  II 

JTIOS  CLASS  2  TERMINAL  PILOT  EVALUATION 

2.1  INTRODUCTION. 

a.  This  section  presents  a  brief  background  of  the 

software  supportablllty  evaluation,  the  evolution  of  the 
User/Supporter  Baseline  Estimate,  the  evaluation  procedure,  results 
of  the  evaluation,  analysis  of  the  results  and  some  lessons  learned 
for  future  application  of  the  RAMSS. 

b.  Part  of  the  pilot  study  task  to  apply  the  Risk  Assessment 

Methodology  for  Software  Supportablllty  (RAMSS)  was  to  develop  the 
details  of  the  proposed  Software  Life  Cycle  Process  (SLCP),  evaluate 
the  SLCP,  and  report  the  lessons  learned  from 

the  evaluation  effort.  The  refined  procedures  for  the  SLCP  evalua¬ 
tion,  evaluation  questions,  source  of  evaluation  question  require¬ 
ment,  and  guidelines  for  the  evaluation  response  are  found  In  the 
Software  Supportablllty  Risk  Assessment  Evaluation  Adaption  Guide¬ 
lines  (reference  1.4.7). 

2.2  BACKGROUND. 

a.  The  was  the  system  selected  by  AFOTEC 

for  a  pilot  study  on  applying  the  Risk  Assessment  Methodology  for 
Software  Supportablllty  (RAMSS).  This  methodology  Integrates  evalua¬ 
tion  data  on  the  software  product,  software  support  resources,  and 
software  life  cycle  process  to  derive  the  risk  to  the  Air  Force  of 
supporting  the  software.  The  software  product  and  software  support 
resources  evaluation  methodologies  already  are  In  use  by  AFOTEC  (see 
reference  1.4.8).  Details  of  the  software  life  cycle  process  (SLCP) 
evaluation  methodology  were  derived  as  part  of  the  current  support 
task  SS  412  (see  reference  1.4.7). 
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b.  m  software  supportabil ity  evaluations 
for  IOTM^were  conducted  at  Eglin  AFB  during  the  period  January  6-17, 
1986.  Prior  to  these  evaluations,  plans  and  preparations  were 
performed  by  AFOTEC  and  the  BOM  Corporation  SS  412  personnel.  In 
addition,  several  meetings  were  held  with  AFOTEC,  WR-ALC,  and  HQ-TAC 
personnel  to  establish  an  initial  User/Supporter  Baseline  Estimate  on 
the  software  support  concept  and  change  profile. 

c.  The  software  product  evaluation  was  conducted  January  6-17 
using  four  evaluators  from  WR-ALC  . ,  and  one 
evaluator  from  TAWC/  ,  The  AFOTEC  Software  Test  Manager  (STM)  and 
Deputy  for  Software  Evaluation  (DSE)  provided  appropriate  calibration 
assistance.  The  software  support  resources  evaluation  was  conducted 
on  January  9  and  10  using  the  five  evaluators  of  the  software  product 
as  well  as  one  BOM  representative.  The  software  life  cycle  process 
evaluation  was  conducted  by  the  8DM  representative  in  parallel  with 
the  software  support  resources  evaluation. 

2.3  EVOLUTION  OF  THE  USER/SUPPORTER  BASELINE  ESTIMATE. 

a.  One  important  part  of  the  pilot  study  was  to  determine  the 
effort  and  procedures  required  to  obtain  a  user/supporter  baseline 
estimate  of  the  '  software  support  concept  and 
change  profile  over  the  first  few  block  releases.  Another  important 
part  was  to  determine  the  impact  of  using  (and  not  using)  such  an 
estimate  during  the  software  supportabil ity  evaluation. 

b.  The  user/supporter  baseline  estimate  is  simply  an  estimation 
of  the  support  resources  and  software  change  activity  expected  for  a 
given  software  system  for  one  or  more  block  releases  during  post 
deployment  software  support.  This  "estimate"  or  "concept"  is  derived 
by  reviewing  historical  software  maintenance  data,  available  acquisi¬ 
tion  planning  information  in  documents  such  as  the  CRISP  or  0/S  CMP, 
the  current  software  system  status  (e.g.,  maturity,  current  develop¬ 
ment  and  support  change  activity),  and  the  views  of  the  using  and 
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supporting  command  personnel.  The  process  may  iterate  until  a 
reasoned®  consensus  or  compromise  is  derived.  This  user/supporter 
baseline-  estimate  then  becomes  a  baseline  against  which  software 
supportabllity  evaluation  measures  can  be  derived  and  the  software's 
supportability  risk  computed. 


c.  Evolution  of  the  .  user/supporter  baseline 
estimate  is  described  in  the  next  several  paragraphs.  Use  of  the 
estimate  is  described  in  later  sections  and  the  conclusions  are 
summarized  in  section  2.7. 


2.3.1  The  User /Supporter  Baseline  Estimate  Evolution  Process.  The 
user/supporter  baseline  estimate  evolved  as  a  series  of  interface 
discussions  among  the  using  command  (HQ-TAC)  representatives,  the 
supporting  command  (WR-ALC)  representatives,  and  AFOTEC/BDM  repre¬ 
sentatives.  The  sequence  of  events  is  listed  below: 

(1)  Visit  with  HQ-TAC  by  AFOTEC/BDM  (November  14,  1985) 

(2)  Visit  with  WR-ALC  by  AFOTEC/BDM  (November  15,  1985) 

(3)  First  draft  generated  by  BDM  (November  22,  1985) 

(4)  Review  of  first  draft  by  HQ-TAC  and  WR-ALC  personnel 
( December  5,  1985) 

(5)  Compromise  for  second  draft  distributed  to  HQ-TAC/WR-ALC 
(December  10,  1985) 

(6)  Final  revision  of  estimate  generated  during  evaluation 
(January  9,  1986) . 
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2.3.2  First  Step:  Visit  with  Using  Command  (HQ-TAC). 

a.  Using  command  personnel  from  HQ-TAC/  provided  valuable 
insight  into  the  derivation  of  support  requirements  and  the  need  for 
those  requirements  from  the  user  perspective.  Several  Issues  from 
the  current  CRISP  were  discussed  and  the  concept  of  the  user/ 
supporter  baseline  estimate  was  presented. 

b.  From  the  user  perspective,  a  new  version  of 

software  will  be  released  to  the  user  approximately  every  6  months. 
The  user  thus  will  have  one  version  in  the  field  and  at  least  one 
version  undergoing  system  field  test  prior  to  being  released  to  the 
field.  The  6-month  "cycle"  is  based  upon  the  Philo¬ 

sophy  and  the  current  6-month  release  cycle  for  the 
Equipment  (  )  which  will  eventually  replace  the  ■ 

system.  There  would  be  some  overlap  in  the  6-month  release  cycle 

workload  for  support  personnel  (see  the  WR-ALC  discussion  for  more 
detailed  Information  on  the  release  cycle). 

c.  The  personnel  allocation  in  the  draft  CRISP,  Volume  III, 

January  1985,  included  a  general  manning  level  of  16  persons  and  an 
additional  5  persons  dedicated  to  specific  software 

support.  The  CRISP  was  not  specific  as  to  all  the  software  which 
will  be  supported,  nor  the  distribution  of  the  16  and  5  persons 
across  the  software  systems.  The  using  command  personnel  could  not 
offer  any  further  clarification  other  than  to  indicate  five  persons 
were  not  enough.  The  latest  draft  CRISP  (November  1985)  has  even 
removed  specific  personnel  requirements  such  as  the  number  and 
function. 

d.  Potential  areas  which  will  increase  the  software  supportabil- 

ity  risk  of  the  include: 

I 

) 

> 

1 

\ 

I 
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Emergency  changes 

' 

4T-.  ..  * 

(2)  Operational  Interfaces  ] 

! 

i 

(3)  Incomplete  Development  (23  of  88  messages  have  not  been  j 

defined) 

(4)  Six-Month  Release  Cycle  (must  be  achieved  to  meet 
releases) 

(5)  Lack  of  personnel  dedicated  to  the  soft¬ 

ware  support. 

e.  The  using  command  personnel  suggested  that  AFOTEC  put  a  form 
of  the  user/supporter  baseline  estimate  in  the  Test  Plan  Outline 
(TPO)  and  evolve  the  user/supporter  baseline  estimate  information 
along  with  the  TPO.  They  also  suggested  there  would  need  to  be  clear 
guidance  (e.g,  authority)  in  order  to  require  such  an  estimate  and 
require  the  maintenance  data  collection  necessary  to  keep  the 
historical  data  base  up  to  date.  Further  discussions  with  AFOTEC 
have  indicated  that  the  CRISP  or  the  Computer  Resources  Life  Cycle 
Management  Plan  (CRLCMP)  may  be  better  documents  to  contain  the 
user/supporter  baseline  estimate. 

2.3.3  Second  Step:  Visit  with  Supporting  Command  (WR-ALC). 

a.  Supporting  command  personnel  from  WR-ALC/  and  WR-ALC/ 

provided  valuable  insight  into  the  current  support  of  the 
and  the  requirements  for 

software  support.  The  currently  estimated  block  release  cycle, 
support  personnel  levels,  and  software  systems  to  be  supported  as 
described  in  the  draft  CRISP,  Volume  III,  January  1985,  were 

discussed.  Guidance  useful  for  deriving  the  necessary  user/supporter 
;  baseline  estimate  was  obtained. 

i 
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b.  Thr  software  systems  to  be  supported  as  part  of  the 
support  Include: 


(1)  Network  Interface  Control  Program  (NICP) 


(2)  Subscriber  Interface  Control  Program  (SICP) 


(3)  Indicator  Control  Panel  Computer  Program  (ICPCP) 


(4)  Plain  Text  Processor  (PTP) 


(5)  Cipher  Text  Processor  (CTP) 


(6)  Integrated  Support  Facility  (ISF)  Support  Software. 


The  ICPCP  software  development  Is  the  responsibility  of  the  Army. 
All  other  software  systems  are  the  development  responsibility  of  the 
Air  Force. 


c.  The  support  release  cycle  will  actually  be  about  9  months  with 
a  3-month  overlay  for  analysis  of  release  change  content.  This  will 
result  in  a  new  version  being  released  to  the  user  approximately 
every  6  months.  The  full  9  months  is  for  engineering  support  and 
does  not  include  technical  orders,  prom  burning,  and  final  field 
distribution  and  test. 


d.  The  actual  personnel  being  used  for  the  soft¬ 
ware  support  includes  nine  persons  for  analysis,  design/implementa¬ 
tion,  and  test,  and  six  persons  for  ISF  support.  The  personnel  have 
skill/experience  levels  ranging  from  2  to  8  years  and  are  evenly 
distributed  across  the  RAMSS  personnel  skill  levels  of  2,  3,  4.  Note 
that  the  RAMSS  uses  five  personnel  skill  levels,  from  1  (lowest, 
meaning  entry  level  personnel)  to  5  (highest,  meaning  the  most 
skilled  and  experienced  personnel).  It  is  expected  that  three  of  the 
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nine  (p^jjp&one  other  person)  will  have  similar 
support^43§bct1ons  and  two  of  the  six  persons  will  have  similar  ISF 

support  functions. 

e.  is  allocated  a  total  of  approximately  43  support 

personnel  for  all  functions  (including  software  support).  The 
formula  for  determining  resource  allocations  was  thought  to  be  in  a 
Joint  Logistic  Decision  Tree  Analysis  AFLC  manual.  Support  command 
personnel  agreed  the  current  CRISP  allocation  of  16  general  support 
personnel  and  5  support  personnel  was  not  suffi¬ 

cient.  At  least  double  the  allocation  of  five  is  required. 

f.  It  is  perceived  that  there  is  lower  support  risk  from  having  a 
single  developer  (rather  than  multiple  developers  as  for  the 

).  Also,  the  Integrated  Support  Facility  (ISF)  is  considered 
to  be  low  risk.  Experience  of  support  personnel  on  the 

indicates  it  takes  much  more  time  than  anticipated  for 
customizing  contractor  procedures  to  the  support  facility,  and 
repeating  contractor  qualification  tests  following  initial  software 
support  changes.  The  learning  curve  progressed  slower  than  expected 
for  these  activities. 

g.  Support  command  personnel  discussed  several  areas  of  concern 
which  would  increase  the  software  supportabllity  risk: 

(1)  Plain  Text  Processor  (PTP)  -  this  firmware  (card  A1  of 

the  Digital  Oata  Processor)  is  not  documented  and  could 

be  subject  to  change 

(2)  Cipher  Text  Processor  (CTP)  -  this  firmware  (card  A6  of 

the  Digital  Data  Processor)  is  not  documented  and  could 

be  subject  to  change 

(3)  Poor  documentation  of  the  operational  software 
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(4)  Security  -  code  Is  classified  as  confidential  and  will 
cause  delays  in  support  response  due  to  a  more  secure 
support  environment  and  security  procedures 

(5)  Unique  Test  Boxes  -  some  one-of-a-kind  test  support  boxes 

developed  by  the  contractor  may  become  an 

availability/upgradeability  problem  during  support 

(6)  Test  Tricks  -  procedures  for  some  known  test  tricks 
(e.g.,  EPROM  write  patch,  6-second  reset  switch)  may  not 
be  documented. 

h.  On  the  basis  of  discussions  with  the  support  personnel,  an 
initial  allocation  of  personnel  was  defined  for  input  to  the  first 

draft  of  the  user/supporter  baseline  estimate.  The  initial  alloca¬ 
tion  Is  shown  in  table  2-1.  Support  personnel  also  provided  an 
understanding  of  the  anticipated  block  release  schedule  (see 

figure  2-1). 


Table  2-1. 


Initial  Support  Personnel  Allocation 


FUNCTION 

NUMBER  OF 
PERSONNEL 

NUMBER  IN  EACH 
SKILL  LEVEL  (1.. 5) 

(1  ■  LOWEST. 

S«  HIGHEST) 

%  DEDICATED 
CLASS  2 

%  DEDICATED 
RELEASE  1 

Ganaral  Support 

1C 

(2J.4.4J) 

10% 

83% 

Oadkatod 

Support  (MO) 

10 

(UJJJ) 

30% 

83% 

(SIO) 

10 

(U4^4I 

30% 

83% 

OOO) 

10 

(UJ24I 

20% 

83% 

(OtfMr) 

10 

<UJ,2,2> 

15% 

83% 

Extarnal  Syttam 

Support  (5%) 

10 

(UJ^I 

0% 

0% 

at-olM-nrwoi 

1.  An  estimate  of  change  requests  for  the  various  software 


systems  for  the  first  few  block  releases  was  not  obtained  directly. 
However,  some  estimates  on  maximums  for  different  categories  were 
done  (see  .table  2-2).  These  estimates  along  with  the  general 
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distribution  of  changes  across  the  categories  were  used  to  obtain  an 
initial  fjfaft  for  the  user/supporter  baseline  estimate.  The  "other" 
category  includes  ISF,  PTP,  and  CTP  software/f irmware  support. 


3  MO. 


START 


6  MO. 


h 


START 


Figure  2-1. 


9  MO. 

RELEASE  1 
3  MO. 


6  MO. 

~  I  — 


Block  Release  Schedule 


9  MO. 

RELEASE  2.  •• 


TTt-O-t 


Table  2-2. 

Estimates  of  Software  Change  Requests 


TYPE 

COMPLEXITY 

PRIORITY 

sw 

SYSTEM 

BLOCK 

RELEASE 

TOTAL# 

CHANGES 

#C 

#H 

#v 

#H 

#M 

#L 

#E 

#u 

#N 

nkp 

1 

mm 

0 

nj 

0 

mm 

mm 

n| 

0 

9 

2 

■S 

2 

1 

i 

11 

3 

mm 

2 

i 

1 

Hi 

B 

i 

i 

13 

SlCP 

1 

9 

9 

0 

mm 

0 

mm 

nj 

mm 

0 

9 

2 

12 

10 

2 

1 

SB 

1 

11 

3 

IS 

12 

2 

KB 

1 

HI 

B 

i 

1 

13 

OCR 

1 

s 

5 

0 

mm 

mm 

KB 

KB 

HI 

0 

5 

2 

« 

5 

1 

if 

ii 

1 

5 

3 

• 

5 

2 

i 

i 

B 

Hi 

i 

1 

6 

Ott»*r 

1 

3 

3 

a 

n 

mm 

0 

mm 

mm 

0 

S 

4 

i 

1 

1 

3 

• 

S 

2 

B 

i 

2 

B 

1 

1 

Hi 

TOTALS 

» 

2« 

MM 

mm 

jn 

0 

MM 

mm 

mm 

26 

(all  CUM  2 

2 

35 

11 

1M 

31 

S-W) 

3 

a 

34 

KB 

KB 

MM 

14 

HI 

MM 

B 

39 

TYRE: 

COMPLEXITY : 

PRIORITY : 

C  ■  CORRECTION 

H  ■  HIGH 

E  -  EMERGENCY 

H  .  ENHANCEMENT 

M-  MEDIUM 

U-  URGENT 

V  «  CONVERSION 

L  -LOW 

N  >  NORMAL 
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2.3.4  Third  Step:  Generate  First  Draft  of  Estimate. 

a.  The  first  draft  of  the  user/supporter  baseline  estimate  was 
generated  from  information  gathered  in  steps  1  and  2  as  well  as  the 
maintenance  release  data  base  (RAMSS  User's  Handbook,  see 
reference  1.4.6).  Ordinarily,  this  estimate  could  have  been  derived 
using  the  RAMSS  automated  support  tool,  but  the  tool  was  not  avail¬ 
able  when  the  first  draft  was  generated.  A  first  draft  of  the 
estimate  could  also  have  been  derived  from  the  maintenance  release 
data  base  prior  to  the  visits  and  then  used  during  the  visits  as  a 
starting  point  for  discussion. 

b.  Another  interesting  feature  which  was  not  available  during  the  - 

generation  of  the  estimate  is  the  estimation  of  the  required  person 
months  per  change  from  a  regression  equation  using  independent  para¬ 
meters  such  as  average  skill  level,  type/complexity/priority  of  base¬ 
line  profile  change  requests,  and  the  type  of  software  system  (e.g. 
Communications-Electronics) .  See  section  3.5  for  more  details  on 
this  feature.  For  purposes  of  future  comparison,  this  feature  is 

integrated  on  each  report  of  the  baseline  estimate  which  displays 
estimated  risk  (the  report  is  a  product  of  the  RAMSS  automated 
support  tool). 

c.  The  first  draft  user/supporter  baseline  estimate  results  are 
shown  in  figure  2-2,  which  is  a  report  generated  by  the  RAMSS  automated 
support  tool.  The  evolution  of  the  release  schedule  and  support 
staff  fro*  the  visits  with  HQ-TAC  and  WR-AIC  personnel  is  apparent. 
The  baseline  support  profile  summarizes  the  changes  across  all  soft¬ 
ware  systems  which  are  to  be  supported.  The  available  person  months 
per  change  is  simply  the  full  time  equivalent  person  months  (computed 
by  using  the  percent  dedicated  and  release  overlap  information) 
divided  by  the  total  number  of  changes.  The  estimated  person  months 
per  change  is  derived  from  a  regression  equation  and  represents  a 
more  realistic  estimate  of  the  optimum  required  resources  based  upon 
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Figure  2-2.  First  Draft  of  Pilot  Study  User/Supporter  Baseline  Estimate 
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the  skill  of  personnel  and  the  workload  complexity  of  the  task. 
There  Is  risk  for  any  level  of  person  months  per  change,  and  this 
estimated  risk  Is  shown  for  each  of  the  first  three  block  releases. 
The  estimated  risk  (see  Glossary  for  precise  definition)  represents 
the  likelihood  that  resources  (personnel  and  schedule)  will  not  be 
adequate  to  meet  the  particular  baseline  change  profile  block  work¬ 
load.  The  estimated  risk  is  computed  by  integrating  over  a  normal 
distribution  with  mean  (the  estimated  person  months  per  change)  and 
standard  deviation  (standard  estimate  of  error  from  the  regression 
equation)  from  the  available  person  months  per  change  to  infinity. 


d.  Optimum  utilization  of  resources  occurs  when  the  available  and 
estimated  persons  months  per  change  are  the  same.  This  does  not  mean 
the  estimated  supportability  risk  is  minimal,  or  even  low.  There  is 
a  tradeoff  between  lowering  risk  by  increasing  resources  and  having  a 
more  optimal  (for  the  estimated  workload)  level  of  resources. 


2.3.5  Fourth  Step:  Review  of  First  Draft  by  HQ-TAC  and 
wft-AlC  Personnel.  “  *  a 


a.  The  Computer  Resources  Working  Group  (CRWG)  for  the 

met  on  December  4-5,  1985,  at  AFOTEC.  It  was 

intended  that  BOM  personnel  would  discuss  the  draft 
user/supporter  baseline  estimate  for  the  software  risk  assessment 
method  shortly  following  the  CRWG,  since  both  the  using  and 
supporting  commands  had  planned  to  attend  the  meeting.  The 

supporting  command  representatives  from  WR-ALC  were  present,  however 
a  last  minute  schedule  change  for  the  using  command  representatives 
In  HQ-TAC  at  Langley  AFB  prevented  their  attendance.  Rather  than 
miss  the  opportunity  to  establish  an  update  to  the  user/supporter 
baseline  estimate,  a  conference  telephone  call  was  performed  with  the 
required  HQ-TAC  and  WR-ALC  participants,  beginning  at  11:30  AM  on 
December  5,  1985,  and  lasting  for  about  1  hour. 


11-12 
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b.  Prior  to  the  telephone  conversation,  inputs  on  the  first  draft 
were  madft  by  the  WR-ALC  personnel.  These  inputs  were  discussed  with 
the  HQ-TAC  personnel  during  the  telephone  conversation. 

c.  The  purpose  of  the  telephone  conversation  was  to  obtain  agree¬ 
ment  on  three  basic  issues: 

(1)  The  number  of  support  personnel  required  to  maintain  the 

software 

(2)  An  understanding  of,  and  agreement  on,  the  software  block 
release  cycle  for 

(3)  The  total  number  of  anticipated  changes,  grouped  by  type, 
complexity,  and  priority,  for  the  first  three  block 
releases. 

d.  Since  both  the  using  and  supporting  command  representat i ves 
agreed  with  the  first  draft  estimate  on  items  1  and  2  above,  the 
remainder  of  this  section  will  discuss  the  evolution  of  item  3  and 
the  proposed  baseline  support  profile  for  this  data. 

e.  As  shown  in  figure  2-3,  the  total  changes  predicted  by  the 
first  draft  estimate  increased  with  each  block  release.  This  data 
was  not  meant  to  predict  that  the  total  changes  would  increase  with 
all  succeeding  block  releases,  but  was  derived  from  the  knowledge 
that  most  software  maintenance  projects  have  increased  change 
activity  for  a  period  of  time  after  operational  release.  After 
reaching  a  peak,  the  change  activity  normally  decreases  with  time. 
Change  activity  data  in  the  various  categories  (type,  complexity, 
priority)  were  chosen  as  representati ve  of  the  historical  data 
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previously  collected  for  systems  like  *  Change  activity  data 
was  based-  upon  planned  changes  to  all  software,  to  include: 


(1)  Network  Interface  Control  program  (NICP) 

(2)  Subscriber  Interface  Control  Program  (SICP) 

(3)  Indicator  Control  Panel  Computer  Program  (ICPCP) 

(4)  Plain  Text  processor  (PTP) 

(5)  Cipher  Text  processor  (CTP) 

(6)  Integrated  Support  Facility  (ISF)  Support  Software. 


total 

CHANGES 
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J  PRIORITY 
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in 

— 

a 

in 

n 

in 

n 

draft 
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Figure  2-3.  Pilot  Study  Development  of  User/Supporter 
Baseline  Change  Profile 
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f.  ,JK1wdicate4  In  figure  2*3,  the  supporting  command  representa¬ 
tives  Mftlijfctcd  much  less  maintenance  activity  than  the  draft.  Part 
of  this  prediction  was  based  on  the  experience  gained  supporting  the 

,  as  well  as  an  expectation  that  the  PTP,  CTP, 
and  ISF  support  software  will  not  be  maintained  by  WR-ALC  personnel. 
According  to  remarks  made  at  the  CRW6,  changes  made  to  the  PTP  and 
CTP  software  will  probably  be  contracted  out  since  there  is  no  docu¬ 
mentation  for_this  software  unless- ESO  funds  a  documentation  effort. 
Other  significant  predictions  by  the  supporting  command  representa¬ 
tives  included:  (1)  a  peak  of  12  changes  will  occur  in  block  2, 
decreasing  to  9  changes  in  block  3;  (2)  there  will  be  no  conversion 
activity;  (3)  there  will  be  few  medium  or  high  complexity  changes; 
and  (4)  there  will  be  no  urgent  or  emergency  changes. 

g.  The  telephone  conversation  with  the  using  command  at  HQ-TAC 
revealed  a  much  different  opinion.  The  using  command  representatives 
commented  that,  based  on  their  experience,  the  original  draft 
baseline  support  profile  seemed  more  realistic.  In  particular,  they 
contended  that  the  supporting  command  predictions:  (1)  were  much  too 
low  in  total  number  of  expected  changes;  (2)  did  not  anticipate  the 
potential  requirement  for  urgent  or  emergency  changes;  and  (3)  did 
not  account  for  the  historical  information  gathered  on  similar 
systems  regarding  change  activity  distribution.  The  using  command 
did  not  give  specific  predictions  in  each  category,  but  indicated 
that  the  supporting  command  should  seriously  consider  the  using 
command  experience  and  opinion  in  establishing  the  baseline. 
Specific  numbers  were  not  agreed  upon  at  the  end  of  the  conversation. 

► 

h.  Immediately  following  the  telephone  conversation,  discussion 
was  continued  with  the  supporting  command  representatives  regarding 
the  baseline  support  profile.  The  results  of  the  discussion  are 
indicated  under  "COMPROMISE"  in  figure  2-3.  In  general,  the  support¬ 
ing  command  agreed  to  increase  the  predicted  number  of  changes  and  to 
use  historical  data  as  guidelines,  although  not  to  the  extent  of  the 
first  draft  recommendations. 
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1.  Each  of  the  change  profiles  discussed  above  have  associated 
estimated  risks  which  correspond  to  the  projected  software  support 
requirements.  A  summary  of  these  risks  is  shown  in  figure  2-4. 


AVAILABLE  PERSON 
TIME  PER  CHANGE  (mo) 

ESTIMATED  PMPC 

ESTIMATED  RISK 

(PMPC) 

DRAFT 

3.20 

1.98 

0.31 

SLOCK  1 

SUPPORTER 

7.57 

1.98 

0.08 
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3.20 

1.9S 

0.31 
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1.98 

0.14 

ORAFT 

1.90 

2.48 

0.61 

SLOCK 2 
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5.SS 
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0.14 

USER 
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2.48 

0.61 

COMPROMISE 

3.33 

3.11 

0.47 

ORAFT 

1.45 

2.89 

0.77 

BLOCK  3 

SUPPORTER 

7.40 

2.41 

0.12 

USER 

1.4S 

2.89 

0.77 

COMPROMISE 

3.33 

3.57 

0.S3 

M-PltO-T*  W43 


Figure  2-4.  Pilot  Study  Risk  by  Block  Release  for  User/Supporter 
i  Baseline  Change  Profile 

j.  These  risks  represent  the  potential  for  unwanted,  negative 
consequences,  such  as  not  meeting  the  desired  block  release  schedule, 
j  For  example,  in  Block  1  the  estimated  risk  is  .14  (compromise), 

meaning  that  there  is  a  14  percent  chance  that  the  projected  software 
■  support  resources  will  not  be  able  to  support  the  predicted  support 

i 

activity.  The  estimated  risks  are  higher  for  the  successive  block 
releases,  as  shown. 

i 

2.3.6  Fifth  Step;  Compromise  for  Second  Draft.  The  user/supporter 
baseline  estimate  takes  into  account  the  opinions,  experience,  and 
knowledge  of  both  the  using  and  supporting  commands.  The  recommended 
baseline  estimate  (the  second  draft)  is  illustrated  in  figure  2-5. 
Comments  similar  to  those  applied  to  the  first  draft  in  section  2.3.4 
are  applicable.  This  estimate  was  used  as  an  initial  input  to  the 
software  supportabi 1 ity  evaluation  process. 
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f-igure  2-5.  Second  Draft  of  Pilot  Study  User/Supporter  Baseline  Estimate 
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2.3.7  Sixth  Step;  Final  Revision  and  Use  of  the  User/Supporter 
Baselin*  Estimate  During  the  Software  Supportabil ity  Evaluation.  The 
second  draft  of  the  user/supporter  baseline  estimate  was  presented  to 
the  evaluators.  The  evaluators  included  the  same  WR-ALC  supporting 
command  personnel  who  had  commented  on  earlier  draft  versions.  A 
more  thorough  analysis  of  personnel  requirements  had  been  conducted 
by  the  supporting  command  personnel.  The  new  allocation  of  personnel 
is  shown  below: 

(1)  2  persons:  10  percent  dedicated  (supervisors) 

(2)  13  persons:  20  percent  dedicated  (general  support) 

(3)  9  persons:  90  percent  dedicated  (direct  support). 

This  allocation  was  slightly  different  than  the  previously  estab¬ 
lished  allocation  in  the  second  draft  (10.9  full-time  equivalents 
versus  11.1  full-time  equivalents).  The  final  draft  of  the 
user/supporter  baseline  estimate  for  the  is 

illustrated  in  figure  2-6. 

2.4  EVALUATION  PROCEDURE. 

a.  The  evaluation  procedures  for  the  software  product  and  soft¬ 
ware  support  resources  were  essentially  as  directed  by  the 
AFOTECP  800-2  guidelines  (see  reference  1.4.8).  The  primary  change 
to  those  procedures  was  the  discussion  of  the  user/supporter  baseline 
estimate  as  it  had  finally  evolved  (see  section  2.3).  This  discus¬ 
sion  helped  orient  the  evaluators  to  the  estimated  personnel  resource 
requirements  and  change  profile  workload.  This  orientation  was  the 
only  noticeable  use  of  the  baseline  estimate  during  the  evaluation 
process,  but  was  considered  very  helpful  to  the  evaluation  partici¬ 
pants.  Oiscussion  of  individual  evaluation  questions  did  not  involve 
reference  to  the  baseline  estimate. 


THE  BOM  CORPORATION 


BDM/ABQ-86-0360-TR 


b.  Th*  software  product  evaluation  was  independently  conducted  in 
parallel-' with  the  other  software  supportability  evaluations.  In 
preparation  for  the  software  support  resources  (SSR)  and  software 
life  cycle  process  (SLCP)  evaluations,  AFOTEC  and  BDM  participants 
met  for  2  hours  on  January  8  in  order  to  review  the  SSR  evaluation 
questionnaires,  discuss  the  evaluation  rating  criteria,  and  set  a 
schedule  agenda  for  the  SSR  and  SLCP  evaluations.  It  was  decided 
that  a  good  approach  would  be  to  Introduce  the  evaluation  group  to 
the  agenda  and  schedule,  briefly  discuss  the  three  areas  of  SSR 
evaluation,  conduct  the  SSR  evaluation  by  area  (completing  responses 
to  evaluation  questions  after  each  discussion),  and  then  discuss  the 
SLCP  evaluation  elements.  Questionnaires  were  completed,  copies  made 
for  evaluators,  and  Introductory  slides  were  prepared. 

c.  The  SSR  evaluation  was  conducted  according  to  the  prepared 
agenda,  although  the  original  time  schedule  was  lengthened.  The  SSR 
evaluation  was  conducted  at  a  level  slightly  below  the  RAMSS  required 
level  so  the  results  have  been  accumulated  to  arrive  at  the  values 
required  in  the  three  areas  of  personnel,  support  systems,  and 
physical  facilities.  The  evaluators  completed  responses  to  questions 
after  sometimes  lengthy  discussions.  Some  questions  were  combined  to 
provide  proper  balance. 

d.  Each  question  was  answered  relative  to  the  adequacy  of  the 
subject  addressed.  Characteristics  of  "adequacy",  i.e.,  "what  does 
adequate  mean  -  how  does  one  judge  adequacy",  were  also  discussed. 
For  example,  characteristics  such  as  planning,  funding  level,  per¬ 
formance,  documentation,  quantity,  and  so  forth  were  used  as  appro¬ 
priate. 

e.  The  discussions  related  to  the  SSR  evaluation  were  very 
helpful  in  understanding  the  plans  for  and  actual  status  of  the  SSR. 
Generally,  the  CRISP  and  0/S  CMP  are  very  poor  in  describing  this 
information.  There  seems  to  be  much  more  capability  than  is  being 
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reflected  In  these  "planning"  documents,  and  in  effect  the  documents 
are  not  being  used  for  planning  at  all.  A  more  complete  description 
of  the  software  product/software  support  resources  evaluation  process 
and  results  will  be  produced  by  AFOTEC  OT&E  personnel.  Results 
necessary  for  input  to  the  RAMSS  are  described  in  section  2.5. 

f.  An  overview  of  the  two  SLCP  major  factors,  project  management 
and  configuration  management,  was  presented  by  the  BDM  participant. 
These  factors  were  discussed  as  processes  conducted  by  the  three 
activities:  procurement,  development  contractor,  and  operation 
support.  Many  deficiencies,  positive  attributes  and  rationale  were 
discussed.  A  subjective  consensus  rating  was  obtained  from  the 
discussion  participants  in  the  various  subelements  across  the 
activities.  These  ratings,  the  discussion,  and  review  of  docu¬ 
ments  have  been  integrated  to  arrive  at  the  required  SLCP  evaluation 
for  input  to  the  RAMSS  pilot  study.  These  results  are  described  in 
section  2.5. 

g.  It  was  very  apparent  from  the  discussion  that  several  more 
sessions  could  have  been  spent  without  covering  all  the  issues.  Some 
of  the  problems  included  short  procurement  schedule,  lack  of  adequate 
procurement  configuration  management  identification,  initial  lack  of 
coordination  between  contractor  and  procurement  activity,  procurement 
organizational  structure  (lack  of  continuity),  contractor  management 
of  subcontractor,  and  so  forth.  The  contractor  test  strategy  was 
considered  to  be  very  complex,  but  thorough.  Operational/Support 
configuration  management  was  considered  poor. 

h.  Although  this  discussion  was  very  valuable  for  the  SLCP 
evaluation,  the  recommended  procedure  is  for  the  AFOTEC  Software  Test 
Manager  and  Deputy  for  Software  Evaluation  to  accumulate  the  required 
SLCP  data  over  a  longer  period  of  time  from  program  reviews,  resource 
working  groups,  procurement  meetings,  and  so  forth.  There  is  simply 
too  much  Information  which  needs  to  be  integrated  across  all  life 
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cycle  phases  and  all  responsible  activities.  Specific  guidelines  are 
presented  in  the  Software  Life  Cycle  Process  Evaluator's  Guide  in 
reference  1.4.7. 

i.  All  necessary  evaluation  data  were  collected  for  the  RAMSS 
pilot  study,  and  the  required  software 
supportabi 1 ity  evaluation  was  completed  satisfactorily.  More  time 
could  have  been  used  for  both  the  SSR  and  SLCP  discussions. 

2.5  EVALUATION  RESULTS. 

a.  The  evaluation  results  required  for  input  to  the  RAMSS  pilot 
study  include  values  from  the  three  software  supportabi 1 ity  level  1 
criteria  (see  figure  2-7):  software  life  cycle  process,  software 
product  maintainability,  and  software  support  resources.  The  evalua¬ 
tion  results  for  the  software  life  cycle  process  were  derived  by  BDM. 
The  evaluation  results  for  the  software  product  maintainability  and 
the  software  support  resources  were  derived  by  AFOTEC  during  the 
planned  IOT&E  of  the  software. 

b.  The  software  supportabi 1 ity  evaluation  results  (on  a  scale  of 
1.0  (low)  to  6.0  (high))  at  the  lowest  level  3  required  for  input  to  the 
RAMSS  are  shown  in  figure  2-8,  along  with  the  accumulated  average 
values  at  the  level  2  and  level  1  of  the  evaluation  hierarchy.  The 
overall  supportabi 1 i ty  score  and  evaluated  risk  are  also  shown.  The 
overall  supportabi 1 ity  score  is  the  unweighted  average  of  the  level  1 
results.  The  evaluated  risk  is  on  a  scale  of  0.0  (no  risk)  to  1.0 
(absolute  certainty).  See  Glossary  for  more  information  on  evaluated 
risk. 

2.5.1  Software  Life  Cycle  Process  Evaluation  Results.  Detailed 
questions  for  this  evaluation,  as  described  in  the  Software  Life 
Cycle  Process  Evaluator's  Guide  (see  reference  1.4.7),  were  only 
informally  used  since  all  questions  were  not  available  during  the 
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evaluation. and  would  ordinarily  be  answered  over  an  extended  period 
of  Values  shown  In  figure  2-8  were  derived  at  the  level  3 

characteristics  and  accumulated  to  the  higher  levels  In  the  usual 
fashion  of  averaging  the  associated  immediate  lower  level  values.  No 
weighting  Is  used  in  the  RAMSS  computations  although  the  level  3 
Inputs  could  have  been  weighted  prior  to  their  Input  to  RAMSS. 

2. 5. 1.1  Software  Project  Management  Evaluation  Results. 

a.  The  rationale  behind  the  software  project  management  evalua¬ 
tion  results  is  briefly  discussed  in  the  following  paragraphs. 

b.  The  planning  for  the  has  been  charac-  - 

terlzed  by  a  very  unrealistic  procurement  activity  schedule,  lack  of 
TEMP  and  CRISP  as  well-detailed  and  implemented  planning  documents, 
and  a  non-existent  0/S  CMP.  These  plans  and  procedures  are  critical 
for  all  aspects  of  software  supportabllity.  In  addition,  lack  of 
planning  for  computer  resource  acquisition  of  the 

integrated  support  system  was  evident  since  there  were  no 
initial  plans  to  acquire  software  bench,  and  integrated  operational 
test  beds.  This  planning  gap  resulted  in  uncertainty  about  what 
development  support  systems  were  being  acquired  and,  once  determined, 
in  poor  documentation  of  these  support  systems.  Contractor  planning 
has  been  somewhat  better  than  the  procurement  and  support  activities, 
but  changes  in  project  requirements  as  well  as  additional  contract 
workload  have  caused  continual  redirection  of  the  contractor 
resources.  Major  requirement  changes  included  a  bilingual  capability 
and  the  packing  of  messages.  After  being  determined,  the  software 
bench  (COMMANDS)  and  the  operational  integrated  test  bed  (TATP) 
station  software  were  tasked  as  deliverables,  but  the  hardware  was 
not.  The  COMMANDS  and  TATP  stations  were  defined  as  a  required 

development  resource,  but  the  contractor  obtained 
other  contracts  (e.g.,  the  United  Kingdom  )  for 

which  these  resources  (systems  and  personnel)  were  to  be  used.  The 
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resourct^draln  on  the  contractor  resulted  in  the  procuring  activity 
requirement  to  purchase  additional  support  resources.  The  procure¬ 
ment  activity,  contractor  activity,  and  support  activity  were  rated 
3.0,  4.0,  and  3.0  for  an  overall  score  of  3.33  for  project  planning. 

c.  The  procurement  activity  organization  structure  has  had 
numerous  problems  because  of  its  lack  of  centralized  control  and 
personnel  consistency  throughout  the.  project  life  cycle.  Continual 
redirection  results  when  lead  personnel  are  changed  or  the  authority 
for  decision  control  changes  or  does  not  exist.  The  organization 
structure  for  procurement  also  suffered  due  to  the  interservice  par¬ 
ticipation  of  the  Army  and  Air  Force.  The  Army  was  not  involved  in 
the  CRWG  and  initially  wanted  all  reference  to  the  Army  removed  from 
the  CRISP.  After  an  attempt  to  write  a  separate  Army  Computer 
Resources  Management  Plan  (CRMP),  the  Army  again  became  a  participant 
in  the  CRISP.  Neither  the  CRISP  nor  CRWG  (nor  support  resources  in 
general)  received  proper  priority  within  the  procurement  activity 
project  structure  with  the  Air  Force  as  lead  agency  or  the  Army  as 
the  deputy  agency.  As  for  personnel  consistency,  since  the  beginning 
of  full-scale  development  around  1981,  WR-ALC  support  activity  has 
had  three  complete  group  turnovers,  program  management  has  had  three 
changes  in  leadership,  configuration  management  office  has  had  two 
leadership  changes,  the  test  branch  and  budget  have  each  had  three 
leadership  changes,  and  the  logistics  branch  has  had  two  leadership 
changes.  The  contractor  also  had  problems  with  staff  attrition  and 
internal  organizational  relationships  with  quality  assurance  and  its 
subcontractor.  The  latter  quality  assurance  interface  is  now  much 
better.  The  organization  structure  for  the  support  activity  is 
getting  better  over  time  as  more  emphasis  is  put  upon  the  post¬ 
deployment  support.  However,  the  CRISP  and  0/S  CMP  should  reflect 
the  support  activity  organization  structure  and  these  documents  do  a 
very  poor  job  of  documenting  this  structure.  In  fact,  the  latest 
version  of  the  CRISP  had  removed  the  few  specific  requirements  for 
support  personnel,  and  the  0/S  CMP  was  just  being  written.  The 
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procure«fl|fc  activity,  contractor  activity,  and  support  activity  were 
rated  3.0*  4.0,  and  3.0  respectively,  for  an  overall  score  of  3.33 
for  project  organization  structure. 


d.  The  design  methods  of  the  various  activities  were  more  diffi¬ 
cult  to  evaluate  without  actually  looking  at  the  pertinent  design 
documents  and  following  the  various  design  reviews.  However,  some 
observations-  are_  tn.  order.  First,  the  procurement  activities 
original  prototype  design  of  the  was  thought 
to  be  good.  Thus,  early  concept  design  must  have  been  reasonably 
good.  However,  the  full  scale  development  procurement  activity 
caused  some  perturbations  in  the  design  through  changing  requirements 
(bilingual  capability  and  message  packing).  The  initial 
system/segment  specification  is  the  procurement  activity  design  docu¬ 
ment,  and  it  must  have  been  reasonably  adequate  even  though  the 
changing  requirements  have  caused  some  schedule  slippage.  The  con¬ 
tractor  activity  has  good  documented  design  methods  as  evidenced  by 
the  internal  standards,  design  documents  (e.g..  Computer  Program 
Development  Plan),  design  reviews,  test  design,  and  use  of  structured 
design  methods.  The  support  activity  has  no  evidence  of  internal 
standards  manuals,  intent  to  transition  the  contractor  design 
methodology,  or  experience  with  modern  design  methods  such  as  data 
flow,  Yourdon  Hierarchy  Charts,  object-oriented  programming,  or 
design  standards/conventions.  The  procurement  activity,  contractor 
activity,  and  support  activity  were  rated  4.0,  5.0,  and  3.0  respec¬ 
tively,  for  an  overall  score  of  4.0. 


e.  The  implementation  methods  of  the  procurement  activity  were 
poor  since  no  particular  standards  or  requirements  have  been 
rigorously  enforced.  Contractor  QA  and  internal  standards  are  in 
place,  but  only  after  a  struggle.  The  Mitre  Corporation  had  been 
assigned  the  Government  QA  monitor  role  and  the  contractor  had  no 
specifically  required  QA  reporting  functions.  Even  now,  however,  the 
QA  standards  are  not  generally  implemented.  The  inconsistencies 
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between  contractor  standards  as  indicated  in  the  Computer  Program 
Development  Plan  and  the  resulting  products  such  as  the  documentation 
and  source  code  indicate  that  good  design  methods  and  techniques  have 
not  been  implemented  in  the  product  forms.  As  examples,  a  naming 
convention  for  global  and  local  variables  was  established  but  not 
rigorously  followed;  instrumentation  standards  were  defined,  but  not 
followed.  The  contractor  now  has  at  least  one  person  full  time  and 
others  part-time,  performing,  the  QA  function,  for  the 

.  The  contractor  has  had  very  good  configuration  management 
control.  The  contractor  unit  test,  integration,  and  system  imple¬ 
mentation  process  has  been  complex  but  thorough.  At  least  a  year 
slippage  in  schedule  is  due  to  this  thoroughness,  and  perhaps  the 
benefit  will  be  seen  in  the  post -deployment  support.  Support 
activity  implementation  methods  have  not  been  carefully  defined. 
Standards  and  conventions  are  not  defined  to  the  level  of  detail 
necessary.  The  procurement  activity,  contractor  activity,  and 
support  activity  were  rated  3.0,  4.5,  and  3.0,  respectively,  for  an 
overall  score  of  3.5  for  implementation  methods. 

f.  The  test  strategies  for  the  procurement  activity  have  been 
very  poor  from  the  initial  lack  of  insight  and  planning  for  acquisi¬ 
tion  of  adequate  development  test  beds  to  the  current  lack  of  direc¬ 
tion  as  to  where  the  purchased  test  beds  will  actually  be  used  during 
post-deployment  support.  The  current  DT&E  activity  is  continually 
fighting  problems  with  hardware  and  software  in  order  to  stay  up  long 
enough  to  conduct  tests.  It  does  not  appear  that  there  is  a 
coordinated  strategy  between  DT&E  and  OT&E  agencies  to  share  test 
data  and  strategies  to  optimize  resources  and  effectiveness  of  the 
T&E  process.  It  does  not  appear  that  there  is  adequate  joint  service 
coordination  (Army  and  Air  Force)  of  the  test  process.  On  the  bright 
side,  the  contractor  has  apparently  done  a  very  thorough  job  of 
developing  a  phased  test  plan,  test  description,  test  procedures,  and 
conf iguration  control  of  the  tests  as  a  test  suite  to  be  transitioned 
to  the  supporting  agency.  The  FQTs  and  PQTs  executed  by  the 
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contractor  have  been  excellent.  However,  the  thoroughness  of  the 
test  strategy  has  caused  a  significant  schedule  slippage  in  the 
project  (perhaps  a  year).  Some  criticism  of  the  lack  of  priority  in 
testing  critical  items  was  noted.  Since  the  test  process  will  be 
transitioned  to  the  support  activity,  the  benefit  might  be  seen  in 
the  post-deployment  support.  However,  the  support  activity  will 
require  more  qualified  personnel  to  understand  and  configure  control 
such  a  complex  test  strategy..  .  The .supporting  activity  needs  to  make 
specific  arrangements  for  definition  and  acquisition  of  its 

Integrated  Support  Facility  computer  resources 
(e.g.,  the  software  bench,  called  COMMANDS,  and  integrated  test  bed, 
called  TATP).  It  was  not  clear  to  the  supporting  command  personnel 
whether  they  would  receive  one,  two,  or  even  none  of  the  current 
development  test  beds.  Interoperability  requirements  do  not  seem  to 
have  been  adequately  addressed  by  the  support  activity.  The  procure¬ 
ment  activity,  contractor  activity,  and  support  activity  were  rated 
3.0,  4.5,  and  3.5  respectively,  for  an  overall  score  of  3.67  for 
implementation  methods. 

g.  The  project  procurement  activity  interfaces  (external)  have 
been  plagued  by  politics,  lack  of  interservice  coordination,  and 
system  interoperabi 1 ity  requirements.  The  procurement  was  planned  as 
a  joint  service  project,  but  there  appears  to  be  much  independence  on 
the  part  of  the  Air  Force  and  Army  participants.  There  have  also 
been  some  problems  with  the  Joint  Program  Office,  the  Mitre  Corpora¬ 
tion,  and  contractor  interfaces.  The  procurement  activity  interface 
with  the  contractor  has  had  some  problems  in  establishing  a  good 
contractor  QA  program.  The  contractor  has  had  some  problems  with  the 
subcontractor.  The  support  activity  does  not  have  a  good  interface 
definition  with  external  project  QA  or  configuration  management 
elements.  Support  activity  interfaces  with  the  procurement  activity 
have  not  been  adequate  to  resolve  the  issue  of  acquisition  of  the 
test  beds.  The  interfaces  between  the  using  and  supporting  command 
personnel  appear  to  be  improved  through  the  evolution  of  the 


THE  BDM  CORPORATION 


BDM/ABQ-86-Q360-TR 


user/sup(K>rter  baseline  estimate.  The  procurement  activity, 
contractor  activity,  and  support  activity  were  rated  2.0,  4.0,  and 
3.0  respectively,  for  an  overall  score  of  3.0  for  project  (external) 
interfaces. 

2.5. 1.2  Software  Configuration  Management  Evaluation  Results. 

a.  The  .rationale  behind  the  software,  configuration  management 
evaluation  results  is  discussed  in  this  section. 

b.  The  procurement  activity  conf iguration  identification  process 
has  been  very  poor.  In  fact,  the  regulations  (see  AFR  65-3, 
MIL'STD-490,  MIL-STD-483)  requiring  certain  contractor  code  identifi¬ 
cation  characteristics  were  not  enforced  (in  fact  were  apparently 
waived).  The  reason  for  this  waiver  is  that  the  Air  Force '  Computer 
Program  Identification  Number  (CPIN)  assignment  process  is  so 
complex,  antiquated,  and  cumbersome  that  no  one  could  complete  the 
proper  paperwork  to  get  CPINs  assigned.  The  effect  is  still  a 
deficiency  in  the  procurement  activity.  The  contractor  internal  con¬ 
figuration  identification  method  has  been  used.  Generally,  the  pro¬ 
curement  activity  is  responsible  for  assigning  identifiers  to 
software  items  at  the  CSCI  level  or  above,  and  the  contractor  is 
responsible  for  assigning  identifiers  to  software  items  below  the 
CSCI  level.  Apparently,  the  contractor  has  done  an  excellent  job  of 
software  item  identification,  baseline  identification,  and  develop¬ 
mental/interim  baseline  identification  (even  though  the  format  of  the 
Version  Description  Document  is  poor).  In  addition,  contractor 
identification  of  change  requests,  forms,  etc.  was  very  good.  The 
support  activity  configuration  identification  is  supposed  to  be  ove**- 
viewed  in  the  CRISP  and  described  in  detail  in  the  0/S  CMP.  The 
CRISP  was  inadequate  and  the  0/S  CMP  was  just  being  written.  The 
procurement  activity,  contractor  activity,  and  support  activity  were 
rated  2.0,  5.0,  and  3.0  respectively,  for  an  overall  score  of  3.33 
for  software  configuration  identification. 
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c.  procurement  activity  has  generally  relied  on  the  contrac¬ 
tor  for^onf iguration  management,  including  control  of  the  change 
process  and  the  various  baselines.  The  general  requirement  to 
control  multiple  variations  of  the  (e.g. 

and  Army)  makes  procurement  control  a  more  difficult  task.  The  con¬ 
tractor  has  maintained  good  documentation  and  control  board  awareness 
of  change  activity  and  tracking  even  though  the  reporting  aspect  has 
not  been  visible  enough  to.- the  procurement-  activity.  The  support 
activity  has  not  yet  adequately  defined  the  support  configuration 
control  procedures  which  should  be  in  the  0/S  CMP  (being  written  at 
the  time  of  this  report)'.  The  procurement  activity,  contractor 
activity,  and  support  activity  were  rated  3.0,  5.0,  and  2.5  respec¬ 
tively,  for  an  overall  score  of  3.5  for  software  configuration 
control. 


d.  The  procurement  activity,  contractor  activity,  and  support 
activity  were  all  lacking  capability  in  status  accounting.  Lack  of 
automated  tools,  inadequate  procurement  requirements  for  contractor 
status  accounting  data,  inadequate  visibility  of  contractor  status 
accounting  data,  support  activity  reliance  on  the  upper  level  WR-ALC 
tracking  forms  (Form  75),  and  in  general  a  lack  of  attention  to  the 
specification  of  status  data  all  make  this  an  area  of  concern  for 
software  supportability  purposes.  All  activities  were  rated  3.0  for 
software  configuration  status  accounting. 


e.  There  was  very  little  evidence  of  a  configuration  management 
audit  capability.  The  formal  procurement  activity  audits  (FCA.  PCA) 
are  a  for*  of  configuration  audits,  but  the  important  internal  audit 
control  function  for  each  activity  was  limited  to  the  usual  review 
process.  The  support  activity  will  have  a  requirement  to  audit  the 
baselines,  change  process,  status  accounting  procedures,  and  adher¬ 
ence  of  the  configuration  management  process  to  established  standards 
and  conventions.  The  procurement  activity,  contractor  activity,  and 
support  activity  were  rated  2.5,  4.0,  and  2.0  respectively,  for  an 
overall  score  of  2.83  for  configuration  audit/review. 
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2.5.2  Software  Product  Maintainability  Evaluation  Results.  The 

software  documentation  evaluation  results  for  the  average  of  the 
SICP  and  the  NICP  documentation  were  used  as  supplied  by  AFOTEC  per¬ 
sonnel.  The  source  listing  evaluation  results  for  the  average  of 

five  SICP  modules  and  six  NICP  .  modules  were  used  as  supplied  by 

AFOTEC  personnel.  The  leve'  3  raw  score  values  for  modularity,  des¬ 
criptiveness,  consistency,  simplicity,  expandability,  and  instrumen¬ 
tation  were  entered.  -The-level  2  and-  level  1  results  were  averaged 
from  level  3  and  level  2  values,  respectively. 

2.5.3  Software  Support  Resources  Evaluation  Results.  The  software 
support  resources  evaluation  results  using  all  six  evaluators  and 
unweighted  values  were  used  as  supplied  by  AFOTEC  personnel.  The 
level  3  values  for  personnel  management,  host  systems,  general 
facilities,  etc  were  entered.  The  level  2  and  level  1  results  were 
averaged  from  level  3  and  level  2  values,  respectively. 

2.5.4  General  Software  Supportabi lity  Evaluation  Results. 

a.  Reference  figure  2-8  for  the  following  discussions.  Three 
values  in  the  figure  refer  to  general  software  supportabi lity  evalua¬ 
tion  results.  The  computed  overall  score  of  3.80  is  the  average  of 
the  level  1  values:  software  life  cycle  process,  software  product 
maintainability,  and  software  support  resources. 


b.  The  Software  Supportabi 1 ity  Confidence  Assessment  is  a  value 
between  0  (low)  and  1  (high)  which  reflects  the  Software  Test 
Manager/Deputy  for  Software  Evaluation  overall  assessment  of  the 
evaluated  software's  supportabi lity.  In  this  pilot  study,  the  BDM 
participant  has  completed  this  assessment  and  assigned  a  value  of 
0.70.  This  important  value  is  only  used  during  the  evaluated  risk 
regression  equation  update  process  (see  the  RAMSS  User's  Handbook, 
reference  1.4.6);  it  does  not  affect  results  of  the  current 
evaluation. 
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c.  evaluated  software  supportabi 1 ity  risk  value  is  computed 
from  tNi  five  scores  for  software  life  cycle  management,  software 
product  maintainability,  personnel,  support  systems,  and  facilities, 
using  a  regression  equation.  The  value  of  0.55  represents  the  pre¬ 
diction  that  55  percent  of  the  block  releases  based  on  a  workload 
estimate  similar  to  the  user/supporter  baseline  estimate  cannot  be 
completed  without  a  workload  or  resource  modification. 

2.5.5.  Software  Supportabi 1 ity  Risk  Assessment. 

a.  The  software  supportability  evaluated  risk  is  derived  from  the 
software  supportabi 1 ity  evaluation  scores  for  software  life  cycle 
process,  software  product  maintainability,  support  resource  person¬ 
nel,  support  resource  systems  and  support  resource  facilities.  A 
regression  equation  (see  RAMSS  User's  Handbook,  reference  1.4.6  and 
section  3.3)  is  used  to  evaluate  the  risk  from  these  five  evaluation 
scores. 


b.  The  plots  of  the  cumulative  distribution  function  for  each 
baseline  estimate  block  release  with  various  risk  assessment  values 
are  shown  in  figures  2-9,  2-10,  and  2-11.  An  overall  summary  of  the 

software  supportabi 1 ity  risk  assessment  is 

shown  in  figure  2-12. 

c.  Based  on  a  risk  scale  of  0.0  to  1.0  (with  low  risk  values 
<  0.20,  medium  risk  values  >  0.20  and  <  0.50,  and  high  risk  values 
>  0.50)  the  primary  results  of  the  risk  assessment  indicate  that  the 

..  overall  evaluated  software  supportability  risk  is  high,  with  the 
primary  risk  drivers  being  the  software  life  cycle  process  and,  at  a 
lower  level,  the  software  configuration  management  process.  Further 
analysis  of  these  results  is  presented  in  section  2.6. 
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2.6  AHttfSIS  OF  EVALUATION  RESULTS. 

This  analysis  will  primarily  be  restricted  to  interpretation 
of  the  risk  assessment  reports  generated  by  the  RAMSS  automated  tool. 
These  reports  are  as  follows:  A3:  Major  Factor  Percentile  Chart, 
A4:  Major  Factor  Risk  Reduction  Chart,  A5:  Plot  of  Person  Months  per 
Change  Versus  Risk,  and  A6:  Summary  of  RAMSS  Results. 

2.6.1  Report  A3:  Pilot  Study  Major  Factor  Percentile  Chart. 

a.  The  results  of  this  report  are  shown  in  figures  2-13  (all 
systems)  and  2-14  (C-E  systems).  Each  of  the  major  factors  and 
level  1  criteria  are  compared  with  the  distribution  of  all  systems. 
The  indicated  percentages  provide  a  relative  understanding  of  how 
well  the  evaluated  software  compared  with  all  other  software  systems. 
The  higher  the  percentages,  the  better  the  evaluated  software  is 
relative  to- other  software  systems. 

b.  The  percentile  chart  indicates  the  software  life  cycle  process 
(the  software  configuration  management  factor),  was  relatively  the 
worst.  The  software  product  (the  source  listings  factor)  was  the 
best.  As  a  guideline,  if  the  score  is  >  0.75,  it  is  high;  if  it  is 
below  0.25,  it  is  low.  Deficiencies  might  be  noted  for  scores  below 
0.25. 


2.6.2  Report  A4:  Pilot  Study  Major  Factor  Risk  Reduction  Chart. 

a.  The  results  of  this  report  are  shown  in  figure  2-15.  The 
major  factor  and  criteria  risk  impact  is  determined  by  computing  the 
difference  in  the  evaluated  risk  for  the  actual  versus  a  perfect 
score.  This  difference  is  then  plotted  as  in  figure  2-15.  The 
interpretation  of  these  differences  is  that  they  represent  the  amount 
the  evaluated  risk  could  be  reduced  if  the  given  factor  or  criteria 
were  to  be  improved  as  much  as  possible  (in  other  words,  given  a 
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perfect.' score  of  6.0).  Thus,  the  factor  or  criteria  which  could  have 
the  mostr  impact  on  reduction  of  risk  will  have  the  longest  plot  of 
asterisks.  The  plot  of  figure  2-15  thus  gives  a  ranking  of  all  the 
major  factors  and  criteria  according  to  impact  on  the  risk  reduction 
of  software  supportability  for  the  Software 
(actually,  only  the  evaluated  NICP  and  SICP  software).  The  risk 
values  are  computed  from  the  evaluated  risk  regression  equation. 


'A 


a 


b.  Of  course,  there  will  be  tradeoffs  as  to  which  factors  and 
criteria  with  the  most  risk  impact  are  viable  candidates  for  improve¬ 
ment,  and  which  lower-level  characteristics  of  those  viable  candi¬ 
dates  can  be  improved.  The  course  of  action  would  be  to  look  at  the 
low  scoring  characteristics  for  each  viable  candidate  factor/ 
criteria,  estimate  a  reasonable  improvement  in  the  score,  recompute 
the  factor/criteria  score,  and  enter  into  the  risk  regression  equa¬ 
tion  to  determine  the  amount  of  reduction  in  the  risk.  This  process 
may  have  to  be  reaccomplished  several  times  to  determine  the  optimum 
tradeoff  benefits. 

c.  From  figure  2-15,  the  candidates  for  possible  risk  reduction 
are  the  software  configuration  management  factor  and  the  software 
project  management  factor.  Focusing  on  the  software  configuration 
management  major  factor,  the  question  is  whether  there  are  lower- 
level  characteristics  which  can  be  improved,  and  if  so,  by  how  much. 
Noting  that  the  is  in  the  fifth  year  of  what 
appears  to  be  a  10-year  full  scale  development  effort  (with  PMRT  in 
1990),  It  appears  there  is  still  plenty  of  time  to  effect  significant 
improvement  in  all  three  activities:  procurement,  contractor,  and 
support. 

d.  Major  improvements  in  the  procurement  activity  would  occur  if: 
current  procedures  would  be  documented;  the  baseline  identification 
function  were  to  be  properly  managed;  automated  tool  support  were 
implemented  which  could  be  transitioned  to  the  support  activity;  and 
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significant  planning  and  preparation  for  product  acceptance  through 
the  FCA  and  PCA  for  the  product  baseline  were  accomplished.  Improve¬ 
ment  in  the  procurement  activity  from  2.63  to  4.25  is  possible.  The 
contractor  could  improve  some  by  automating  status  accounting  func¬ 
tions  and  improving  the  audit/review  process.  Not  much  improvement 
is  anticipated.  The  contractor  activity  might  improve  from  4.25  to 
4.5.  The  support  activity  has  the  best  chance  of  improvement  by: 
developing  an  agreed  on  and  high  quality  0/S  CMP;  updating  the  CRISP 
to  include  proper  high  level  detail  consistent  with  the  0/S  CMP; 
adopting  rigorous  internal  procedures  and  standards;  writing  and 
adhering  to  an  internal  configuration  management  plan;  and  working 
with  the  procurement  activity  to  effect  an  automated  configuration 
management  system  for  baseline  software  documentation  and  code 
control,  and  change  status  accounting  and  reporting.  Improvement  in 
the  support  activity  from  2.63  to  5.0  is  possible.  Overall  this 
represents  an  improvement  in  the  configuration  management  major 
factor  from  3.17  to  4.58.  This  would  represent  a  reduction  in 
overall  evaluated  software  supportabi 1 ity  risk  from  0.55  to  0.43. 

2.6.3  Report  A5:  Pilot  Study  Plot  of  Cumulative  Distribution 
Risk  Function. 

a.  The  results  of  this  report  are  shown  in  figures  2-9,  2-10,  and 
2-11.  The  primary  use  of  these  plots  is  to  visually  display  the 
software  supportabi 1 ity  risk  versus  the  workload  of  available  person 
months  per  change  (PMPC).  The  two  figures  are  plots  for  each  base¬ 
line  estimate  block  release  for  the  evaluated  system.  From  these 
plots  the  analyst  can  get  a  feel  for  the  variances  between  the  risk 
(estimated  and  evaluated),  and  among  the  person  months  per  change 
workloads  (available,  estimated,  evaluated).  See  the  Glossary  for 
definitions  of  key  terms  used  above. 


b.  If  the  evaluated  risk  is  lower  than  the  estimated  risk,  that 
says  the  quality  of  the  life  cycle  process,  software  products,  and 
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support  Resources  may  overcome  some  of  the  estimated  risk  based  only 
'  on  worteiSid  and  expected  change  profile.  If  both  evaluated  and 

[  estimated  risk  are  low  (e.g.,  below  0.2),  then  the  software  supporta- 

bility  risk  for  the  system  can  be  considered  low.  If  both  evaluated 
and  estimated  risk  are  high  (e.g.,  above  0.5),  then  the  software 
\  supportability  risk  for  the  system  can  be  considered  to  be  high. 

i 

i 

c.  The.  analyst  can.  perform  "what-if*  sensitivity  analysis  with 
the  cumulative  distribution  plots  and  the  risk  values.  For  example, 
the  evaluated  risk  is  derived  from  the  software  supportability 
evaluation  scores.  Those  scores  were  based  on  the  baseline  estimate. 
If  the  baseline  estimate  were  to  be  changed  (i.e.,  increase  person¬ 
nel,  decrease  change  profile  workload),  then  theoretically  the  soft¬ 
ware  supportability  evaluation  would  have  to  be  conducted  again 
against  the  new  baseline.  Since  this  is  not  practical,  an  alternate 
approximation  is  suggested.  Using  the  cumulative  distribution  plot 
(e.g.,  figure  2-10  for  block  release  2),  the  analyst  should  plot  the 
evaluated  risk  value  (0.55)  and  determine  the  evaluated  person  months 
per  change  (approximately  2.76).  Then,  use  the  RAMSS  tool  to 
construct  a  new  cumulative  distribution  function  based  upon  the  new 
block  release  2  baseline.  Using  the  old  2.76  evaluated  person  months 
per  change  plotted  against  the  new  cumulative  distribution  function 
would  give  the  approximate  evaluated  risk  against  the  new  baseline 
estimate.  The  change  in  the  evaluated  risk  (reduction  in  this  case) 
could  then  be  noted  for  sensitivity  purposes  and  risk  reduction 
analysis. 

2.6.4  Report  A6:  Pilot  Study  Summary  of  RAMSS  Results. 

a.  The  results  of  this  report  are  shown  in  figure  2-11.  This 
report  presents  a  concise  summary  of  the  evaluation  scores,  major 
factor  risk  impact,  estimated  and  evaluated  risk,  and  ratings  of  the 
software  supportabi 1 ity  risk  values  as  HIGH  (above  threshold),  MEDIUM 


s 


1 


a 
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(thresh®|j|jto  goal),  or  LOW  (below  goal).  The  threshold  and  goal 
values  a^psubjectlve  at  this  time,  but  represent: 

(1)  Threshold  *  0.5  means  50  percent  of  the  time  the 

user/supporter  baseline  workload  estimate  will 
not  be  met. 

(2)  Goal  3  0.2  means  20  percent  of  the  time  the  user/ 

supporter  baseline  workload  estimate  will  not 
be  met. 

b.  Even  with  these  values,  it  does  not  necessarily  mean  that 
missing  the  estimate  will  have  much  impact.  If  a  scheduled  release 
misses  by  a  few  days,  it  may  or  may  not  be  a  large  impact.  No  risk 
impact  functions  (more  correctly  called  utility  functions)  have  been 
derived  as  part  of  this  effort.  However,  if  one  were  to  consider 
negative  events  to  be  normally  distributed  with  catastrophic  and  very 
minor  impact  as  the  respective  boundary  conditions,  then  some 
estimate  of  mean  and  standard  deviation  could  be  projected  for  each 
specific  risk  agent  and  a  utility  function  generated.  For  example, 
this  might  mean  that  given  a  risk  of  0.4,  only  1  percent  of  the 
negative  events  are  catastrophic  or  with  a  risk  of  0.5  as  many  as 
25  percent  of  the  negative  events  are  catastrophic.  The  investiga¬ 
tion  of  utility  functions  is  beyond  the  scope  of  the  current  analysis 
effort. 

2.7  LESSONS  LEARNED. 

a.  The  following  list  summarizes  the  lessons  learned  during  the 
pilot  study  application  of  RAMSS  to  the  soft¬ 

ware. 

(1)  The  user/supporter  baseline  estimate  (USBE)  was  able  to 
be  derived,  but  required  some  reasonable  "guesses"  based 
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upon  maintenance  release  data.  The  RAMSS  support  tool 
Mill  help  this  process,  but  some  continual  training  of 
AFOTEC  personnel  will  be  required. 

(2)  The  main  benefit  of  the  USBE  was  in  stimulating  the 
interaction  and  discussion  between  using  and  supporting 
commands,  and  within  the  using  command.  It  is  recom¬ 
mended  that  the  USBE,  in  agreement  form,  be  present  in  a 
software  support  document  (TPO,  CRISP,  or  CRLCMP). 

(3)  The  USBE  was  not  a  major  factor  in  answering  the 
individual  evaluation  questions. 

(4)  The  SLCP  evaluation  cannot  be  done  in  the  same  manner  as 
the  other  evaluations.  For  credibility,  it  Is  essential 
to  capture  the  life  cycle  process  characteristics  over 
time  and  create  an  historical  base  upon  which  the  SLCP 
questions  can  be  answered.  However,  it  should  be 
possible  to  interact  with  system  experts  during  the  life 
cycle  to  capture  this  history  using  the  SLCP  questions 
(see  reference  1.4.7)  as  a  checklist. 

(5)  The  use  of  the  RAMSS  tool  will  aid  the  interpretation  of 
the  risk  assessment  results.  There  are  several  "what  if" 
functions  that  can  be  done.  For  example,  in  trying  to 
pinpoint  specific  characteristic  risk  reduction,  it  is 
possible  to  determine  the  "new"  risk  given  several 
"temporary"  changes  to  characteristics  scores  (see 
report  A2  discussion  in  appendix  B). 

(6)  One  important  side  effect  of  the  USBE  evolution  process 
is  the  data  gathered  from  both  using  and  supporting 
command  personnel  concerning  areas  of  risk.  These  areas 
of  risk  can  be  investigated  by  AFOTEC  personnel  for 
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potential  impact  upon  the  system  OT&E  as  well  as  the 
software  portion  of  the  OT&E. 

(7)  The  using  and  supporting  command  personnel  were  very 
cooperative  during  the  pilot  study.  They  seemed  to 
appreciate  the  opportunity  to  participate  in  the  specifi¬ 
cation  of  the  USBE.  The  using  command  had  some  reserva¬ 
tions  initially  about  AFOTEC's  role  in  this  area. 
Reluctance  to  "sign-up"  to  agreed  on  USBE  values  may 
occur  for  some  systems. 

(8)  The  development  effort  has  several 

"generic"  life  cycle  process  flaws  which  have  been 
observed  across  many  systems: 

a)  The  full  scale  development  schedule  of  27  months 
defined  in  1980  was  much  too  ambitious.  Current 
projections  are  for  PMRT  in  1990. 

b)  Functional  expectations  changed  from  the  prototype 
demonstration. 

c)  Interoperability  requirements  with  other  services 
were  (are)  a  source  of  problems. 

d)  Planning  for  computer  support  resources  during  the 
post-deployment  phase  has  been  very  poor.  Generally, 
very  little  priority  is  given  to  this  function. 

e)  Organizational  centralization  of  responsibility  and 
consistency  of  personnel  over  the  project  life  cycle 
has  been  poorly  managed. 

f)  Configuration  management  plans,  procedures,  and 

automated  tool  support  are  inadequate. 
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g)  Security  concerns  (e.g.,  classified  software  docu- 
>  mentation  and  source  listings)  have  not  been  properly 

addressed.  Solution  is  frequently  to  declassify 
information  (perhaps  arbitrarily). 

h)  Procurement  activity  understanding  of  the  deliverable 
requirements,  as  reflected  in  the  RFP/CDRL/etc. ,  has 
been  inadequate  in  the  area  of  computer  resources, 
test  support,  and  quality  assurance. 

b.  There  is  a  significant  amount  of  project  management  and  con¬ 
figuration  management  which  is  being  done,  but  not  being  properly 
incorporated  into  the  proper  planning,  specifications,  and  other 
documents.  For  example,  the  support  activity 

personnel  knew  much  more  information  concerning  the  plans,  organiza¬ 
tion  structure,  test  strategies,  configuration  control,  personnel 
allocations,  and  facility  layout  than  was  contained  in  the  TEMP, 
CRISP,  or  0/S  CMP.  Apparently,  these  documents  are  not  having  as 
much  effective  use  as  is  possible. 


III.  Refinements  ta  RAMSS 

♦  V 


Effl 
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SECTION  III 


REFINEMENTS  TO  RAMSS 


3.1  INTRODUCTION. 

The  baseline  for  the  RAMSS  Is  contained  In  volumes  1  and  2  of 
the  report  "Software  Supportablllty  Risk  Assessment  In  OT&E: 
Historical  Baselines  for  Risk  Profiles,"  (see  reference  1.4.5). 
There  have  been  several  refinements  to  the  methodology,  procedures, 
and  statistical  analysis  results  represented  In  that  baseline  report. 
This  section  discusses  the  major  refinements  In  some  detail.  The 
complete  RAMSS  Evaluator's  Guide,  which  Integrates  these  refinements. 
Is  presented  In  appendix  B  of  this  document. 

3.2  TRANSFORMATION  OF  HISTORICAL  EVALUATION  DATA. 

a.  The  RAMSS  baseline  report  (reference  1.4.5)  contained  a  data 
base  of  evaluation  on  81  software  systems  using  the  new  software 
supportablllty  hierarchy  across  the  three  criteria:  software  life 
cycle  process,  software  product  maintainability,  and  software  support 
resources.  The  hierarchy  evaluation  scores  were  obtained  at  each  of 
the  levels  (3,  2,  1,  0)  of  the  hierarchy  (reference  figure  2-7). 

b.  For  purposes  of  continued  application  of  this  historical  data 
for  future  AFOTEC  evaluations,  a  new  evaluation  data  base  has  been 
generated  using  the  level  3  evaluation  values.  The  values  were  con¬ 
verted  frovi  -50  to  +50  scale,  to  the  AFOTEC  1-6  scale.  The  level  3 
values  were  averaged  to  obtain  level  2  values  and  so  forth  until 
evaluation  scores  at  all  levels  were  computed.  The  old  evaluation 
data  base  level  0  score  (ASUPPORT  for  overall  software  supportabll- 
Ity)  has  been  transformed  to  a  new  variable  ACONFID  which  has  a  value 
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between  0  (low)  and  1  (high)  representing  the  overall  software 
supportafrtllty  confidence  rating.  The  transformation  equation  Is: 

ACONFIO  -  50  * 

The  old  data  base  risk  var1ab1e»  ARISK,  was  determined  to  be  too 
erratic  for  further  use.  The  concept  of  “confidence"  seemed  to  be 
easier  for  an  evaluator  to  assess  consistently  than  that  of  "risk", 
even  though  for  our  purposes  the  general  software  supportablllty  risk 
would  be  simply  1-ACONFID.  It  Is  this  risk  value  against  which  the 
software  supportablllty  evaluated  risk  regression  equation  has  been 
derived  (see  section  3.4)  using  the  evaluation  factors  described  In 
section  3.3.  This  use  of  ACONFIO  In  the  derived  risk  regression 
equation  Is  the  only  use  of  this  variable,  but  It  enables  AFOTEC  to 
maintain  a  reasonably  current  equation  for  computing  the  evaluated 
software  supportablllty  risk  for  an  evaluated  software  system  using 
an  extensive  historical  data  base.  As  the  data  base  evolves  and 
becomes  more  accurate  and  larger,  the  regression  equation  should  be 
more  accurate  as  well. 

c.  Further  specification  of  the  historical  evaluation  data  as  It 
has  been  transformed  Is  described  In  the  RAMSS  User's  Handbook 
(reference  1.4.6)  along  with  all  the  data  base  Information  which  is  a 
part  of  this  risk  assessment  methodology. 

3.3  NEW  SUPPORT ABILITY  FACTORS. 

a.  The  RAMSS  baseline  report  (reference  1.4.5)  presented  an 
analysis  approach  (In  section  4.4.2. 1)  to  determine  the  grouping 
relationship  of  the  44  supportablllty  rating  variables  used  on  the 
data  collection  survey  form.  This  analysis  approach  Is  called  factor 
analysis.  After  studying  the  data,  7  variables  were  eliminated  and 
the  remaining  37  were  used.  The  factor  loadings  resulting  from  that 
analysis  are  shown  in  figure  3-1.  The  Interpretation  of  those 
results  Is  shown  In  table  3-1. 
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R0TAT8B  FACTOR  COAOIAOS 


FACTO* 

FACTOR 

FACTOR 

FACTOR 

FACTOR 

FACTOR 

1 

-2 

3 

4 

< 

6 

afooc  3 

0.131 

C.742* 

0.011 

0.349 

0.207 

0.2C2 

afoocfcO  * 

0.181 

C.S41* 

X  .  231 

0.434  • 

0.019 

0.012 

I’oacecs  s 

0.211 

C  .721  • 

-0.002 

0.024 

0.217 

0.112 

afooccon  4 

0.201 

0.311* 

0.244 

0.422 

0.204 

-0.161 
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Figure  3-1.  Baseline  Report  Factor  Analysis  Results 

b.  The  factor  analysis  results  were  very  encouraging  since  all 
the  factors  except  "organization"  were  elements  in  AFOTEC's  software 
supportablllty  evaluation  hierarchy.  The  "organization"  factor  seems 
to  cross  several  of  the  hierarchy  elements.  These  factor  results 
were  satisfactory  enough  to  consider  using  a  five-factor  model  (all 
but  the  "organization"  factor)  for  regression  analysis.  Computa¬ 
tional  complexity  of  the  factor  analysis  model  justified  a 
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slmpllftaftJon  of  the  five-factor  model  so  that  the  factor  values  are 
simply  tip  average  of  the  factor's  lower  level  characteristic  values. 
Thus*  tlM^JMv  software  supportablllty  factors  for  use  In  regression 
analysis  are  shown  In  table  3-2,  and  the  factor  values  are  obtained 
from  a  simple  cumulative  average  rather  than  by  the  complicated 
factor  analysis  computation.  The  significant  factor  analysis  values 
(Indicated  by  an  asterisk  In  figure  3-1)  justify  the  use  of  the 
evaluation  hierarchy  characteristic  values. 


Table  3-1. 

Baseline  Report  Supportablllty  Factors 


Table  3-2. 

New  Software  Supportablllty  Factors 


M-01M-TO.W4S 


III-4 
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3.4  REGfSSSION  EQUATION  FOR  EVALUATED  SUPPORTABILITY  RISK. 

a.  Thm  mathematical  model  for  regression  analysis  Is 

T  ■  Uq  +  *  •  *  *  +  ^5^5  +  ® 

where 

Y  «  the  transformed  general  software  supportablllty  risk 
rating 

X|  -  the  score  for  the  1th  factor 
e  ■  a  random  component 

and  the  regression  coefficients  Uq,  b^,  ...»  bg  are  parameters  to  be 
estimated.  The  Xj  values  are  computed  for  the  five  factors  described 
In  section  3.3.  The  general  software  supportablllty  risk  ratting  Is 
derived  from  the  historical  evaluation  data  base  variable  ACONFID. 
The  AC0NFI0  value  Is  the  evaluator's  overall  confidence  In  the  sup¬ 
portablllty  of  the  software  based  upon  all  possible  software  support- 
ability  factors  and  a  baseline  estimate  of  the  software  change 
profile.  The  value  Is  obtained  from  an  evaluator  (probably  the  Soft¬ 
ware  Test  Manager)  Independently  of  the  other  software  supportablllty 
evaluations. 

b.  The  computation  of  the  evaluated  software  supportablllty  risk 
thus  follows  from  the  mathematical  model: 

R  -  risk  »  1 -ACONFID  0  <  R  <  I 

R  •  predicted  risk  (evaluated  software  supportablllty  risk) 
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where 

L  *  4n<rHr) 

-  bQ  ♦  b^APROOUCT)  +  b2(AEPER)  ♦  b3(AESYS) 

+  b4(AEFAC)  +  b5(AMANAGE) 

The  constants  b^  are  the  regression  equation  coefficients. 

c.  The  BMOP  (reference  1.4.10)  statistical  regression  package 
results  for  this  model  are  shown  In  figure  3-2.  The  equation  for  L 
thus  becomes: 

L~  -  4.90401  -  0.29131  (APRODUCT)  -  0.15600  (AEPER) 

-  0.25120  (AESYS)  +  0.04294  (AEFAC) 

-  0.66174  (AMANAGE) 


PAGE  3  BMDP1R  L(RISK)  VS.  SUPPORTABILITY  FACTORS 
REGRESSION  TITLE  IS 
L<RISK>  vs.  suppqrtability  factors 
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Figure  3-2.  Results  for  Regression  Analysis  (Transformed) 
Risk  Versus  Supportablllty  Factors 


d.  The  sequence  of  computations  to  determine  the  evaluated  soft¬ 
ware  supportablllty  risk  given  the  evaluation  values  for  APRODUCT, 
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AEPER,  AESYS,  AEFAC,  and  AMANAGE  Is  Illustrated  by  the  following 

examples*?)^ 

- 

Suppose: 

APROOUCT  (Product  Maintainability)  -  4.15 
AEPER  (Personnel  Resources)  ■  3.53 
AESYS  (System  Resources)  *  3.72 
AEFAC  (Facility  Resources)  »  4.58 
AMANAGE  (Life  Cycle  Process)  -  3.32 

then: 

l~  -  4.90401  -  0.29131  (4.15)  -  0.15600  (3.53) 

-  0.25120  (3.72)  +  0.04294  (4.58) 

-  0.66174  (3.32) 

*  0.20962 

,  1  .02 
i  +  e-0. 20962  ~  T 

-  0.55 

e.  The  only  anomalous  aspect  of  this  regression  equation  Is  the 
plus  sign  of  the  AEFAC  coefficient.  This  would  seem  to  Imply  that 
the  better  the  AEFAC  the  higher  the  risk.  Actually,  the  AEFAC  factor 
is  not  significant,  as  can  be  determined  from  the  P(2  TAIL)  column  In 
figure  3-2.  The  AEFAC  coefficient,  even  though  It  Is  positive.  Is 
very  small.  The  AEFAC  factor  Is  retained  to  maintain  parallelism 
with  the  AFOTEC  software  supportablllty  evaluation  hierarchy.  The 
regression  equation  will  evolve  over  time  as  more  evaluations  are 
performed  by  AFOTEC,  and  data  Is  added  to  the  historical  evaluation 
data  base. 

3.5  COMPUTATION  OF  SUPPORTABILITY  FACTOR  RISK  REDUCTION. 


(1  -  .02)"1 


a.  An  Important  aspect  of  risk  analysis  Is  to  determine  which 
software  supportablllty  crlterla/major  factors  have  the  most  "Impact" 
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upon  ttoft!*.  A  simple  method  was  derived  to  obtain  some  perspec¬ 
tive  of  figt  maximum  possible  reduction  In  evaluated  software  support- 
ability  riat  due  to  each  of  the  software  supportablllty  criteria  and 
Major  factors.  Those  crlterla/major  factors  which  could  potentially 
effect  the  most  reduction  In  risk  could  be  considered  to  be  the  risk 
"drivers." 


b.  The  method  of  computing  the  risk  reduction  Is  as  follows: 

(1)  Compute  the  evaluated  software  supportablllty  risk 

from  the  regression  equation  using  the  five  factor 
evaluation  scores. 

(2)  For  each  software  supportablllty  criterion  and  major 

factor,  determine  the  risk  under  the  assumption  of 

maximum  crlterlon/major  factor  Improvement  (l.e.,  an 

evaluation  score  of  6.0). 

(3)  Compute  the  difference  In  risk  R£  -  R1  for  each  cri¬ 

teria/factor  1. 

(4)  The  criteria/factors  with  the  largest  computed  differ¬ 
ence,  l.e.,  largest  potential  reduction  In  risk,  are 
considered  to  be  the  risk  drivers. 

c.  As  an  example,  the  data  from  the  example  In  section  3.4 
resulted  In  an  evaluated  software  supportablllty  risk  of  0.55.  The 
following  potential  risk  reductions  are  computed: 


APROOUCT  (4.15  -  6.00) 

AEPER  (3.53  -  6.00) 

AESYS  (3.72  -  6.00) 

AEFAC  (4.58  -  6.00) 

AMANAGE  (3.32  -  6.00) 

SUPPORT  RESOURCES  (3.94  -  6.00) 


Risk 

Reduction 

at 

0.55  - 

0.42 

* 

0.13 

Risk 

Reduction 

3 

0.55  - 

0.46 

3 

0.09 

Risk 

Reduction 

3 

0.55  - 

0.41 

3 

0.14 

Risk 

Reduction 

3 

0.55  - 

0.57 

3 

-0.02 

Risk 

Reduction 

3 

0.55  - 

0.17 

3 

0.38 

Risk 

Reduction 

3 

0.55  - 

0.33 

3 

0.22 
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Thm*m  clearly  show  that  the  AMANAGE  (Life  Cycle  Process) 
crltari^^Ml*  aoat  Impact,  followed  by  SUPPORT  RESOURCES,  and  then 
APfMOUCl^jFlffr  a:  similar  manner,  the  major  factor  components  of 
AMAHAfit  and  APROOUCT  can  be  Included  to  determine  the  potential 
reduction  In  risk  due  to  an  improvement  In  those  major  factors.  The 
results  can  be  compared  against  the  AEPER,  AESYS,  and  AEFAC  risk 
reduction  results  to  determine  major  factor  risk  drivers. 

d.  The  RAMSS  analysis  report  "A4:  MAJOR  FACTOR  RISK  REDUCTION 
CHART"  gives  a  pictorial  representation  of  the  potential  risk 
reduction  results.  Examples  of  this  report  are  in  section  2, 
appendix  B,  and  In  the  RAMSS  User's  Handbook  (reference  1.4.6). 
Further  analysis  can  be  easily  conducted  to  determine  which 
crlterla/major  factors/characteristics  cap  feasibly  be  Improved 
across  the  evaluation  scores.  The  potential  reduction  of  the 
evaluated  supportabl  1 1  ty  risk  can  then  be  computed.  For  the  example 
above,  suppose  such  an  analysis  were  to  focus  upon  the  software  life 
cycle  process  (AMANAGE),  since  this  factor  appears  to  have  the  most 
Impact,  and  It  were  determined  that  a  realistic  Improvement  In  the 
characteristics  would  raise  the  evaluation  score  from  3.32  to  4.58. 
Then  the  evaluated  software  supportablllty  risk  would  drop  from  0.55 
to  0.35,  a  substantial  reduction.  Further  analysis  may  be  performed 
by  the  STM/OSE  to  Identify  other  areas  where  improvement  would 
significantly  raise  the  overall  evaluation  score. 

3.6  REGRESSION  EQUATION  FOR  BASELINE  ESTIMATED  WORKLOAD. 

a.  During  the  analysis  effort  for  the  RAMSS  baseline  report 
(referencm  1.4.5)  there  was  not  sufficient  time  to  determine  whether 
Important  relationships  existed  among  the  data  collected  for  the 
maintenance  releases.  In  particular,  it  was  anticipated  that  the 
resources  expended  in  person  months  for  a  given  release  might  be 
dependent  upon:  the  skill  level  of  the  personnel;  the  distribution 
of  changes  across  type,  complexity,  and  priority;  and  the  functional 
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nature  software.  Other  parameters  such  as  number  of  source 

lines  (tlRI^  and  percentage  of  high  level  language  source  lines 
(total)  also  affect  the  resources  expended.  Parameters  such  as 

the  percentage  of  source  lines  changed  and  percentage  of  changed 
source  lines  which  are  high  level  language  could  not  be  collected  as 
part  of  the  data  collection  effort. 

b.  A  regression  analysis  using  a  model  similar  to  the  model 
described  in  section  3.4  has  been  done,  and  the  results  are  signifi¬ 
cant  enough  to  Incorporate  Into  the  RAMSS.  The  following  regression 
equations  have  been  derived  to  predict  the  person  months  per  change 
workload  required  for  a  given  profile  of  change  requests  on  a  system 
whose  software  Is  of  type  OFP,  CE,  EH,  ATD,  ATE,  SIM,  or  SUP,  and 
whose  support  personnel  have  a  certain  average  skill  level.  The 
historical  data  Is  reasonably  accurate,  but  there  Is  hope  that 
Improved  maintenance  release  data  In  the  future  might  Improve  our 
understanding  of  these  macro  relationships. 

c.  The  regression  model  equations  are: 

PMPC  »  Person  months  per  change  request  workload 

PMPC  *  predicted  workload 

PMPC~  *  eL 

where 

L~  *  bQ  +  b^AVGSKILL)  +  b2(PTC0RR)  +  b3(PCL0W) 

+  b4(PCHIGH)  +  bg(PPNORM) 

10 

.  I  b, (TYPE,) 

1»6 

AVGSKILL  *  average  skill  (1  -low  to  5-h 1 gh)  of  support 

personnel 

PTCORR  *  percentage  of  change  requests  which  are  corrections 

PCIOW  *  percentage  of  change  requests  which  are  low 

complexity 


iir-io 
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percentage  of  change  requests  which  are  high 
complexity 

percentage  of  change  requests  which  are  normal 
priority 

1  If  system  type  Is  same  as  TYPE^ 

0  otherwise 


Five  types  have  explicit  coefficients.  The  INTERCEPT  constant  Is  the 
Implicit  coefficient  for  both  SUP  and  SIM.  The  results  of  the 
regression  analysis  Including  the  coefficients  are  shown  In 
figure  3-3.  The  significant  variables  are  PCHIGH,  PPNORM,  AVGSKILL, 
and  In  some  respect,  the  system  functional  type  (ATD,  ATE,  ...). 
Because  the.  addition  of  more  maintenance  release  data  ta  the 
historical  maintenance  release  data  base  could  provide  new  Insights 
Into  the  parameters,  the  complete  set  of  parameters  shown  In 
figure  3-3  will  be  retained  In  this  Initial  regression  model. 
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Figure  3-3.  Results  for  Regression  Analysis  (Transformed) 

Person  Months  Per  Change  Versus  Maintenance  Release 
Profile  Oata 
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d.  Immorder  to  obtain  the  estimated  risk,  a  refinement  of  the 
techn1qu0$md  In  the  RAMSS  baseline  report  (reference  1.4.5)  has 
been  derived.  In  the  baseline  report  a  simple  counting  technique  was 
employed  to  determine  the  estimated  risk.  The  historical  sample  of 
number  of  systems  releases  within  a  certain  range  of  person  months 
per  change  was  viewed  as  a  probability  density  function.  The 
estimated  risk  for  the  system  being  evaluated  was  then  computed  as 
the  area  under  the  curve  greater  than  or  equal  to  the  available 
person  months  per  change  as  computed  from  the  user/supporter  baseline 
estimate.  Two  distributions  were  of  Interest:  all  system  release 
data  points,  and  all  system  release  data  points  for  systems  of  the 
same  type  as  the  system  being  evaluated.  Thus,  there  would  be  two 
estimated  risks  which  could  be  computed.  The  two  distributions 
approximated  a  normal  distribution  with  some  universal  mean  and 
standard  deviation. 

e.  From  the  regression  equation  for  the  estimated  person  months 
per  change,  a  more  accurate  refinement  of  the  computation  for 
estimated  risk  can  be  derived  In  which  the  covariates  of  the  regres¬ 
sion  equation  are  significant.  It  Is  assumed  that  the  historical 
values  for  available  person  months  per  change  are  normally  distrib¬ 
uted  with  mean  (the  estimated  person  months  per  change)  and  variance 
(the  square  of  the  standard  error  of  estimation  from  the  regression 
analysis  Is  a  best  estimate).  This  distribution  of  the  available 

a 

person  months  per  change  (L^)  about  the  regression  line  (l  )  for  one 
of  the  covariates  (X)  Is  Illustrated  In  figure  3-4.  If  all  such 
regression  lines  for  all  covariates  were  flat  (zero  slope),  then  the 
resulting  one  distribution  (shown  on  the  left  of  figure  3-4)  would 
correspond  to  the  baseline  report  historical  sample  distribution  for 
all  systems.  Thus,  this  refinement  results  In  a  family  of  distribu¬ 
tion  functions  oriented  about  the  regression  equation. 
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Figure  3-4.  Illustration  of  Estimated  Risk  Distribution  Functions 

f.  Given  the  situation  shown  In  figure  3-4,  It  becomes  possible 
to  refine  the  current  risk  estimation  procedure  by  taking  advantage 
of  the  regression  model.  Recall,  In  the  old  procedure,  an  available 
PMPC  value,  PMPC^,  Is  compared  to  the  distribution  of  sample  PMPC 
values  across  the  releases  of  all  systems  or  systems  of  one  type,  and 
risk  Is  estimated  as  the  proportion  of  sample  releases  having  PMPC 
values  greater  than  or  equal  to  PMPC^.  This  risk  estimate  Is 
Illustrated  as  the  shaded  area  under  the  distribution  curve  labelled 
D  that  appears  against  the  in { PMPC)  axis  of  figure  3-4. 

g.  Use  of  a  sample  distribution  like  D  to  estimate  risk  is  fine 
In  the  absence  of  a  more  detailed  model  to  represent  PMPC.  Once  a 
more  sophisticated  model  can  be  built,  though,  that  model  should  be 
exploited  to  estimate  risk.  In  the  regression  model  used  here,  PMPC 
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Is  relatflit  to  such  variables  as  proportion  of  high  complexity 

changes;  therefore,  a  risk  estimate  should  also  be  related  to  those 

variables*  Figure  3-4  shows  that  the  regression  approach  may  yield 

very  different  risk  estimates  than  the  sample  distribution  approach. 

Note  that  estimated  risk  (the  shaded  area  under  the  curves  labelled 
*  2 

N(L  ,o  )  depends  on  the  value  of  the  generic  covariate  X.  If  X  »  X,, 
estimated  risk  Is  nearly  1;  If  X  *  X^,  estimated  risk  Is  nearly  0;  If 
X  -  Xg,  risk  Is  1/2,  a  value  close  to  that  estimated  via  the  sample 
distribution.  The  sample  distribution  yields  one  risk  estimate  that 
Is  not  Influenced  by  X  values  and  is  therefore  misleading  In  light  of 
the  regression  model. 


h.  Tables  3-3  and  3-4  compare  risk  estimates  based  on  the  two 
different  methods  for  hypothetical  available  PMPC  values.  In  each 
table,  there  are  three  columns  of  risk  estimates  which  are,  from  left 
to  right,  based  on  the  regression  method,  the  sample  distribution 
over  all  system  types,  and  the  sample  distribution  for  one  system 
type.  These  results  are  derived  from  two  actual  cases  of  data  and 
from  the  regression  model  fitted  to  a  subset  of  the  current  software 
maintenance  block  release  data  set.  Note  In  table  3-4  that  the 
methods  can  Indeed  yield  substantially  different  results. 

Table  3-3. 

Old  and  Refined  Estimated  Risk  Methods  Yield  Similar  Results 


ESTIMATED  In  (PMPC):  .9545  SWTYPE:  C*E 

ESTIMATED  PMPC:  2.5974  CASE  NO.:  104 


AVAILABLE 

RISK 

■Hi# 

rWV 

In  (PMPC) 

REGRESS 

ALL  SYS. 

C'E  SYS. 

i 

0 

.8420 

.871 

.923 

2 

0.6931 

.6083 

.639 

.723 

3 

1.0986 

.4400 

.443 

.490 

4 

1.3863 

.3247 

.346 

.400 

5 

1.6094 

.2454 

.257 

.284 

M-43M.TII-W.07 
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Jfc  Table  3-4. 

0T6  Estimated  Risk  Methods  Yield  Olfferent  Results 

B&ATSO  In  (PMPC):  2.141  SW  TYRE:  EW 

ESTIMATED  PMPC:  3.508  CASE  NO.:  13C 


3.508 


AVAILABLE 

RISK 

PMPC 

In  (PMPC) 

REGRESS 

ALL  SYS. 

C-E  SYS. 

1 

0 

.9879 

.871 

.773 

5 

1.6094 

.7121 

.257 

.273 

10 

2.3026 

.4327 

.064 

.136 

15 

2.7081 

.2753 

.014 

.045 

20 

2.9957 

.1845 

.007 

.000 

1.  In  order  to  compute  the  estimated  risk  using  the  refined 
approach.  It  must  be  possible  to  numerically  estimate  Integration 
over  a  normal  distribution  function  with  a  mean  and  variance.  Recall 
that 

L~  ■  bQ  ♦  bx  (AVGSKILL)  +  b?  (PTCORR)  ♦  b3  (PCLOW) 

♦  b4  (PCHIGH)  +  bg  (PPNORM) 


♦  L  •>i(TYPEi> 

1*6 


PMPC  *  estimated  person  months  per  change 


■  eL  . 


Estimated  risk  for  an  available  PMPC,  PMPC^,  Is  calculated  as 


R  *  Estimated  Risk  =  I  -  F 


m(PMPCA)  -  L 
5 
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where 


FO*' 


cumulative  standard  normal  distribution  function 
evaluated  at  x 

an  estimate  of  the  standard  deviation  of  the  normal 
distribution  of  the  about  L  . 


The  value  for  "s"  Is  obtained  from  the  BHOP-generated  table  of 
regression  results  (see  figure  3-3),  where  It  Is  labelled  "STD.  ERROR 
OF  EST."  and  has  a  value  In  this  case  of  0.9509.  The  function  F  may 
be  numerically  approximated  by 


\  +  S6H(x)j  [j  G  ^)] 

V*  *  a32'  ♦  a42 


SGN(x) 


at  -  0.278393 
a2  *  0.230389 
a3  -  0.000972 
a4  -  0.078108 

1  If  x  >0 
-1  If  x  <  0 


j.  As  an  example,  the  JTIDS  Class  2  Terminal  user/supporter  base¬ 
line  estimate  (see  section  2,  figure  2-6)  has  the  following  block  3 
values  for  the  regression  model  parameters: 

AVGSKILL  -  3.0 
PTCORR  -  0.65 
PC LOW  -  0.65 
PCHIGH  -  0.05 
PPNORM  -  0.65 
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E* 
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Thus,  ^ 

-  0.07083  ♦  (0.54837)  (3.0)  +  (-0.02948)  (.65) 
'  ♦  (0.21815)  (.65)  ♦  (2.09568)  (.05) 

♦  (-1.57228)  (.65)  +  0.35255(1) 

-  1.2739275 
PHP(f  -  3.57 


The  available  PMPC  Is  computed  as 

PMPC  (available)  »  [(15*. 19  ♦  9*.90)*(0.667)*(9.0) |/20 
*  3.285 


The  estlnated  risk  Is  therefore 


.  ^/in(3.285)  -  1.2739275, 

R  ■  1  -  F(—J - - ) 


0.9509 

1  -  F (-0.0889272) 

1 


-  +  SGH(-0. 0889272) 

-  1  -  +  £g (0.06288 10  )j 

-  1  -  ♦  \  (1  *  (1  +  0.0175056  +  0.0009110 

+  0.0000002  +  0. 0000012 )_4| 

■  1  -  +  £  (1  -  0.9295990)j 

i  £  ♦  £(0.0704010) 

-  0.5  +  0.0352005 


0.5352005 
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3.7  S0FniM£  SUPPORTABILITY  EVALUATION. 

a.  Thr  RAMSS  baseline  report  (reference  1.4.5)  provided  an  over¬ 
view  of  the  methodology  and  an  extended  example  of  the  methodology 
application.  For  the  most  part  the  spirit  of  that  Information 
remains  current.  There  are  some  minor  procedural  differences  In  the 
evaluation  process,  use  of  historical  maintenance  profiles,  computa¬ 
tion  of  risk  values,  and  so  forth.  The  major  changes  In  the  software 
supportablllty  evaluation  process  Is  In  the  form  and  use  of  the 
user/supporter  baseline  estimate,  and  In  the  Improvements  of  software 
life  cycle  process  evaluation. 

b.  The  user/supporter  baseline  estimate  Is  recommended  for  use  In 
all  three  software  supportablllty  evaluations:  software  life  cycle 
process,  software  product  maintainability,  and  software  support 
resources.  However,  Its  use  Is  probably  more  Important  as  a 
mechanism  to  stimulate  using  and  supporting  command  discussion 
regarding  the  projected  personnel  resources  and  change  profile 
workload.  The  baseline  estimate  would  be  more  useful  In  this  sense 
during  the  evaluation  calibration  activity  than  through  direct 
reference  during  the  actual  evaluations.  The  RAMSS  still  uses  the 
concept  of  evaluating  against  the  baseline,  so  It  Is  understood  what 
risk  means  (l.e.,  a  negative  event  means  the  baseline  workload 
estimate  was  not  met). 

c.  The  software  life  cycle  characteristics  In  reference  1.4.5 
Inadvertently  did  not  Include  the  Implementation  methods.  Project 
cost  has  been  changed  to  organization  structure,  and  organization 
Interfaces  has  been  changed  to  project  Interfaces.  This  new  termi¬ 
nology  Is  consistent  with  the  data  collected  during  the  generation  of 
historical  maintenance  profiles.  In  addition,  the  configuration 
management  system  characteristic  in  the  software  support  resources/ 
support  systems  hierarchy  was  unintentionally  not  included.  These 
minor  changes  are  more  editorial  in  nature,  but  can  create  confusion 
if  referencing  across  prior  documents. 
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d.  Fejr  purposes  of  consistency*  the  following  reports  should  be 
consIdapiP.taportant  references  for  the  stated  reasons.  All  refine¬ 
ments  of'Hlf  RAMSS  are  In  the  new  baseline  RAMSS  documents. 

(1)  Old  RAMSS  Baseline  Report  (reference  1.4.5).  Use  this 
report  for  background  Into  the  process  through  which  the 
RAMSS  has  been  derived.  All  historical  data,  procedures, 
and  methodology  are  more  accurately  reflected  In  the  new 
baseline  documents. 

(2)  RAMSS  User's  Handbook  (reference  1.4.61.  This  report  Is 
one  of  the  new  baseline  RAMSS  documents.  Use  It  to 
understand  how  to  Implement  a  RAMSS,  obtain  evaluation 
analysis  reports,  and  update/report  contents  of  the 
historical  data  bases  through  the  automated  support 
system  for  RAMSS. 

(3)  RAMSS  Adaptation  Guidelines  (reference  1.4.7).  This 
report  Is  one  of  the  new  baseline  RAMSS  documents.  Use 
It  to  understand  how  to  adapt  the  current  AFOTEC  software 
supportablllty  evaluations  to  the  requirements  of  the 
RAMSS.  In  addition,  an  appendix  contains  the  Software 
Life  Cycle  Process  Evaluator's  Guide. 

(4)  RAMSS  Pilot  Study  and  Methodology  Refinements.  This 
report  Is  one  of  the  new  baseline  RAMSS  documents.  It 
Includes  results  of  a  pilot  study  using  RAMSS,  an  RAMSS 
Evaluator's  Guide  (appendix  B),  and  a  briefing  of  RAMSS 
(appendix  A)  for  general  use  In  describing  the  main 
features  of  the  RAMSS. 

(5)  AFOTECP  800-2  Series.  This  AFOTEC  pamphlet  series  Is 

basic  to  the  RAMSS  since  the  Software  Product  Maintain¬ 
ability  (volume  3)  and  the  Software  Support  Resources 
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4.  (volume  5)  Evaluator's  Guides  are  Included.  Updates  to 
the  pamphlet  series.  Including  Management  of  Software 
Operational  Test  and  Evaluation  (volume  1).  should  care¬ 
fully  consider  Implications  of  the  new  baseline  RAMSS 
documents.  In  particular,  the  RAMSS  Evaluator's  Guide 
and  Software  Life  Cycle  Process  Evaluator's  Guide  were 
written  In  a  form  which  should  be  easily  adaptable  as 
AFQTECP  800-2  volumes. 
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SECTION  IV 

CONCLUSIONS  AND  RECOMMENDATIONS 


4.1  INTRODUCTION. 

This  section  summarizes  some  of  the  more  important  issues 

which  will  affect  the  production  use  of  the  RAMSS  by  AFOTEC  person¬ 
nel.  It  must  be  understood  that  any  risk  assessment  will  only  be  as 

good  as  the  data  upon  which  it  is  based,  and  the  process  through 

which  the  data  are  analyzed  and  conclusions  derived.  Most  of  the 
conclusions  and  recommendations  of  this  report  are  in  some  way 

concerned  with  the  data  collection  or  the  implemented  process  neces¬ 
sary  to  obtain  a  valid  risk  assessment  of  software  supportabi 1 ity. 

4.2  USING  RAMSS  FOR  AFOTEC  PROGRAMS. 

The  following  conclusions/recommendations  have  been  derived 
from  the  process  of  developing  the  RAMSS. 

(1)  The  RAMSS  is  not  mature,  but  it  should  provide  useful 
analysis  results  and  conclusions.  The  RAMSS  must  evolve 
through  application  in  order  to  reach  its  full  potential. 
The  evolution  includes  updating  the  historical  evaluation 
and  maintenance  release  data  and  the  associated  risk 
regression  equations,  and  refining  the  procedural  aspects 
of  applying  the  RAMSS  to  actual  software  assessments. 

(2)  The  User/Supporter  Baseline  Estimate  is  a  useful  mecha¬ 
nism  to  facilitate  using  command,  supporting  command,  and 
AFOTEC  personnel  interaction  concerning  computer  resource 
support  requirements .  This  Estimate  is  valuable  input  to 
the  calibration  briefing/discussion  prior  to  software 
supportabi 1 ity  evaluations.  The  Estimate  has  limited  use 
during  completion  of  software  supportabi 1 ity  evaluation 
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questionnaires.  The  Estimate  can  be  derived  by  AFOTEC 
personnel  using  the  RAMSS  automated  support  system 
independent  of  using/supporting  command  personnel  for  use 
in  early  life  cycle  high  level  software  supportabi 1 ity 
evaluations. 

(3)  The  update  of  the  regression  equations  to  reflect  new 
evaluation  and/or  maintenance  release  data  should  be 
carefully  controlled.  One  or  two  updates  a  year  should 
be  sufficient.  Each  update  should  include  a  thorough 
statistical  analysis  of  the  8MDP  statistical  reports. 
Instructions  for  performing  the  updates  are  contained  in 
the  RAMSS  User's  Handbook  (reference  1.4.6). 

(4)  AFOTEC  should  maintain  a  lessons  learned  history  of  the 
Software  Life  Cycle  Process  evaluations  so  the  procedures 
for  collecting  information  and  mapping  the  information 
into  the  appropriate  questionnaire  responses  can  become 
more  consistent  and  routine. 

(5)  The  RAMSS  historical  data  base  is  partly  subjective. 
Continued  data  collection  should  provide  more  accuracy 
and  maintain  the  currency  of  the  information. 

(6)  It  is  strongly  recommended  that  a  position  of  RAMSS 
system  manager  be  created  and  filled  by  an  AFOTEC  person 
(e.g.f  a  civilian)  who  would  provide  continuity  from 
program  to  program  and  year  to  year.  The  RAMSS  system 
manager  would  provide  expertise  to  the  STM/DSE  for  each 
software  OT&E  effort,  maintain  the  RAMSS  and  supporta- 
bility  evaluation  procedures,  and  operate  the  automated 
support  analysis  tools  (e.g.,  RAMSS,  QAP,  and  ASSET'. 
Perhaps  the  most  important  concern  in  use  of  trie  RAMSS  oy 
AFOTEC  is  its  consistent  application.  With  the  nat-^e 
AFOTEC  personnel's  temporary  duty  assignments. 


HO-AUl  *73  RISK  ASSESSMENT  METHODOLOGY  FOR  SOFTWARE  SUPPORTABILITY 
<RAHSS>:  PILOT  EV.  .  (U>  BOH  CORP  ALBUQUERQUE  HA 
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unlikely  that  a  corporate  knowledge  of  RAMSS  and  the 
software  supportabil ity  evaluations  can  be  maintained 
without  positive  action  to  create  a  permanent  RAMSS 
system  manager  position. 

(7)  The  major  documents  for  understanding  and  using  the  RAMSS 
are: 

a)  Software  Supportabi 1 ity  Risk  Assessment  in  OT&E: 
Historical  Baselines  Risk  Profiles.  BDM/A-85-0510-TR, 
Volumes  I  and  II,  October  7,  1985 

b)  Risk  Assessment  Methodology  for  Software  Supporta¬ 

bil  ity  (RAMSS):  User's  Handbook,  BDM/A-85-1270-TR, 
April  14,  1986 

c)  Risk  Assessment  Methodology  for  Software  Supporta¬ 
bil  ity  (RAMSS):  Guidelines  for  Adapting  Software 

Supportabil ity  Evaluations,  BDM/ABQ-86-0090-TR, 

April  14,  1986 

d)  Software  Life  Cycle  Process  Evaluator's  Guide. 
BDM/ABQ-86-0090-TR,  Appendix  A,  April  14,  1986 

e)  Risk  Assessment  for  Software  Supportabi 1 ity  (RAMSS): 

Pilot  Evaluation  Results  and  Methodology  Refinement, 
BDM/ABQ-86-0360-TR,  April  14,  1986. 

f)  Overview  Briefing  of  RAMSS,  BDM/ABQ-86-0360-TR, 
Appendix  A,  April  14,  1986 

g)  Risk  Assessment  Methodology  for  Software  Supporta¬ 
bil  ity  (RAMSS): _ Evaluator's  Guide.  BDM/ABQ-86- 

0360-TR,  Appendix  B,  April  14,  1986 
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h)  AFOTECP  800-2,  Software  OT&E  Guidelines,  Volume  III, 
Software  Maintainability  Evaluators  Guide,  Volume  V, 
Software  Support  Resource  Evaluation  User's  Guide. 

The  Software  Life  Cycle  Process  Evaluator's  Guide  and  the  RAMSS 
Evaluator's  Guide  should  be  adapted  as  part  of  the  AFOTECP  800-2 
series.  Volume  V  of  the  AFOTECP  800-2  series  is  no  longer  being 
published  and  should  be  appropriately  updated  to  make  the  use  of  the 
ASSET  automated  support  tool  more  effective. 

4.3  PILOT  EVALUATION 

The  following  list  summarizes  the  conclusions/recommendations 
from  the  pilot  study  application  of  RAMSS  to  the 
software. 

(1)  The  user/supporter  baseline  estimate  (USBE)  was  able  to 
be  derived,  but  required  some  reasonable  "guesses"  based 
upon  maintenance  release  data. 

(2)  The  main  benefit  of  the  USBE  was  the  interaction  and 
discussion  among  using  command,  supporting  command,  and 
AFOTEC  personnel . 

(3)  The  USBE  was  not  a  major  factor  in  answering  the 

individual  evaluation  questions. 

(4)  The  Software  Life  Cycle  Process  (SLCP)  evaluation  cannot 

be  done  in  the  same  manner  as  the  other  evaluations.  For 

credibility,  it  is  essential  to  capture  the  life  cycle 

process  characteristics  over  time  to  create  a  "history" 
base  upon  which  responses  to  the  SLCP  questions  can  be 

based. 
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(5)  The  use  of  the  RAMSS  tool  will  aid  the  interpretat ion  of 
the  risk  assessment  results,  but  there  are  several  more 
"what  if"  analysis  functions  that  could  be  developed. 

(6)  One  important  side  effect  of  the  USBE  evolution  process 
is  the  using  and  supporting  command  personnel  identifica¬ 
tion  of  areas  of  risk.  These  areas  of  risk  can  be 
investigated  by  AFOTEC  personnel  for  potential  impact 
upon  the  system  OT&E  as  well  as  the  software  portion  of 
the  OT&E. 

(7)  The  using  and  supporting  command  personnel  were  very 
cooperative  during  the  pilot  study.  They  seemed  to 
appreciate  the  opportunity  to  participate  in  the  specifi¬ 
cation  of  the  USBE. 

(8)  The  development  effort  has  several 

"generic"  life  cycle  process  flaws  which  have  been 
observed  across  many  systems: 

a)  The  full-scale  development  schedule  of  27  months 
defined  in  1980  was  much  too  ambitious.  Current 
projections  are  for  PMRT  in  1990. 

b)  Functional  expectations  changed  from  the  prototype 
demonstration . 

c)  Interoperabi 1 ity  requirements  with  other  services 
were  (are)  a  source  of  problems. 

d)  Planning  for  computer  support  resources  during  the 
post-deployment  phase  has  been  very  poor.  Generally, 
very  little  priority  is  given  to  this  function. 
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e)  Organizational  central ization  of  responsibi 1 ity  and 
consistency  of  personnel  over  the  project  life  cycle 
has  been  poorly  managed. 

f)  Configuration  management  plans,  procedures,  and  auto¬ 
mated  tool  support  are  inadequate. 

g)  Security  concerns  (e.g.,  classified  software  docu¬ 
mentation  and  source  listings)  have  not  been  properly 
addressed. 

h)  Procurement  activity  understanding  of  the  deliverable 
requirements,  as  reflected  in  the  RFP/CDRL/etc. ,  is 
inadequate  in  the  area  of  computer  resources,  test 
support,  and  quality  assurance. 

(9)  There  is  a  significant  amount  of  project  management  and 
configuration  management  which  is  being  done,  but  not 
being  properly  incorporated  into  the  proper  planning, 
specifications,  and  other  documents.  For  example,  the  JTIDS 
Class  2  Terminal  support  personnel  knew  much  more  infor¬ 
mation  concerning  the  plans,  organization  structure,  test 
strategies,  configuration  control,  personnel  allocations, 
and  facility  layout  than  was  contained  in  the  TEMP, 

CRISP,  or  0/S  CMP. 

4.4  DERIVING  A  USER/SUPPORTER  BASELINE  ESTIMATE. 

a.  The  theoretical  basis  of  the  RAMSS  requires  the  use  of  a 

user/supporter  baseline  estimate  of  support  resources  and  workload 
change  profile  in  order  to  have  a  baseline  against  which  a  software 
supportabi 1 ity  evaluation  can  be  conducted.  Thus,  the  measure  of 

risk  derived  from  the  evaluation  scores  is  relative  to  meeting  the 

baseline  workload  estimate.  Without  such  an  estimate,  the  evaluation 
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would  not  have  a  specified  baseline  and  it  would  be  more  difficult  to 
interpret  the  resulting  derived  risk  (i.e.,  what  would  constitute  a 
negative  event?). 

b.  The  requirement  that  using  and  supporting  command  personnel 
arrive  at  a  consensus  on  the  baseline  estimate  is  not  essential.  It 
is  not  even  necessary  that  the  using  and  supporting  command  partici¬ 
pate  in  the  derivation  of  the  baseline  estimate.  The  baseline 
estimate  could  be  derived  by  AFOTEC  personnel  using  the  historical 
maintenance  release  data,  computer  resources  support  planning 
documents,  and  the  RAMSS  automated  support  system.  The  resulting 
baseline  estimate  and  subsequent  software  supportabi 1 ity  risk  assess¬ 
ment  would  be  consistent  and  could  be  appropriately  reported  by 
AFOTEC. 

c.  Although  the  RAMSS  does  not  require  using  and  supporting 
command  personnel  participation,  it  is  highly  recommended.  The 
benefits  of  this  participation  during  the 

Pilot  Evaluation  were  significant.  The  communication  among  using 
command,  supporting  command,  and  AFOTEC  personnel  signif icantly 
improved  the  accuracy  of  the  baseline  estimate.  The  understanding  of 
follow-on  support  requirements  among  the  participants  was  greatly 
improved.  Areas  of  supportabi 1 ity  risk  were  identified  by  both  using 
and  supporting  command  personnel.  Results  of  the  discussions  should 
aid  in  future  updates  to  the  CRISP  and  0/S  CMP. 

d.  The  using  and  supporting  command  personnel  were  very  suppor¬ 
tive,  and  seemed  pleased  to  be  involved  in  the  process  of  deriving  a 
baseline  workload  estimate. 

e.  The  user/supporter  baseline  estimate  derivation  process 
consists  of  four  basic  steps  any  of  which  may  serve  as  the  starting 
point,  and  all  of  which  may  require  some  iteration. 
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(1)  STEP  1:  Oerive  a  draft  user/supporter  baseline  estimate 
from  the  RAMSS  automated  support  tool  entry  and  analysis 
procedures. 

(2)  STEP  2:  Obtain  comments  from  the  supporting  command 

personnel  on  the  draft  user/supporter  baseline  estimate. 

(3)  STEP  3:  Obtain  comments  from  the  using  command  personnel 
on  the  draft  user/supporter  baseline  estimate. 

(4)  STEP  4:  Oerive  a  compromise  from  the  draft  estimate  and 
using/supporting  command  comments. 

The  contact  with  the  using/supporting  command  personnel  can  be 
through  on-site  visits  and/or  telephone  conversations. 

f.  The  user/supporter  baseline  estimate  should  be  discussed 
prior/during  the  software  supportability  evaluations.  The  most 
likely  focus  is  during  the  evaluator  calibrations  for  the  software 
product  and  software  support  resources  evaluations.  The  software 
life  cycle  process  evaluation  is  a  more  long-range  process  in  which 
early  data  collection  will  provide  information  for  the  baseline 
estimate. 

g.  There  was  very  little  opportunity  to  determine  the  effect  of 
not  using  a  baseline  estimate.  The  possibility  of  not  having  using 
and  supporting  command  personnel  participation  has  been  considered 
above.  An  evaluation  could  be  performed  with  no  baseline  estimate, 
but  very  little  additional  information  above  the  evaluations  scores 
could  be  obtained.  In  particular,  there  could  be  no  estimated  risk 
computation  and  the  evaluated  risk  would  have  no  baseline  interpreta¬ 
tion  upon  which  to  interpret  the  meaning  of  the  risk  value.  Since 
AFOTEC  personnel  can  derive  a  baseline  estimate  independent  of  other 
participants,  there  would  seem  to  be  no  good  reason  why  a  baseline 
estimate  could  not  be  derived. 
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4.5  OATA  COLLECTION  TO  UPGRADE  RAMSS. 

a.  The  data  from  future  AFOTEC  software  supportabi 1 ity  evalua¬ 
tions  should  be  entered  into  the  historical  evaluation  data  base. 
Maintaining  these  data  from  actual  evaluations  is  critical  to  the 
evolution  of  the  RAMSS. 

b.  It  is  critical  that  maintenance  release  data  continue  to  be 

collected  and  entered  into  the  historical  maintenance  release  data 
base.  These  data  are  the  basis  for  connecting  actual  maintenance 

activity  with  the  AFOTEC  software  supportability  evaluation  results. 

In  order  for  support  site  personnel  to  obtain  the  necessary  release 
data,  it  is  necessary  to  make  the  data  collection  process  efficient 
and  somewhat  related  to  activity  already  being  accomplished.  A 
recommended  data  collection  form  and  procedure  is  discussed  in 
references  1.4.5  and  1.4.6.  The  essential  elements  of  the  the  form 
and  procedure  are: 

(1)  The  form  and  procedure  are  temporary  until  a  more 
permanent  arrangement  can  be  integrated  into  the  Air 
Force  software  support  concept. 

(2)  All  cognizant  software  support  sites  and  major  (critical) 

software  systems  currently  being  supported  should  be 
solicited  to  participate  in  the  data  collection  effort. 
Initially  it  is  recommended  that  A.-0TEC  contact  personnel 
responsible  for  the  systems  currently  in  the  historical 
data  base  and  request  continued  support  for  the  collec¬ 

tion  of  maintenance  release  data. 

(3)  It  is  estimated  that  completion  of  the  data  collection 

form  (and  altering  current  practices  so  the  data  are 
readily  available)  would  Lake  very  little  additional 
support  personnel  time.  The  range  might  be  from  one 
person  day  to  one  person  week  per  release. 
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(4)  The  data  collection  form  data  elements  required  for  each 

release  include:  site,  system,  software  system,  software 
system  type,  size  in  thousands  of  source  lines,  source 
languages,  personnel  counts  and  skill  levels,  release 
identification,  release  start  date,  engineering  comple¬ 

tion  date,  field  release  date,  and  baseline  software 
support  profile  data  on  each  change  request  in  the 
release. 

(5)  The  data  collection  procedure  would  involve  each  support 
site  completing  a  data  collection  form  for  each  software 
system  release.  The  form  would  be  sent  to  a  data 
repository  site  (AFOTEC,  at  this  time)  for  integration 
into  the  current  data  base,  update  of  the  historical 
maintenance  profiles,  and  further  statistical  analysis. 

(6)  It  is  recommended  that  such  a  data  collection  form  be 

adopted  and  that  AFOTEC  develop  the  necessary  data  base 

and  analysis  environment  to  support  regular  revisions  to 
the  historical  maintenance  profiles.  The  current  RAMSS 
automated  support  system,  (see  RAMSS  User's  Handbook, 
reference  1.4.6)  is  adequate  to  accomplish  this  function. 

4.6  SUMMARY. 

a.  In  summary,  the  RAMSS  should  be  an  effective  tool  for  AFOTEC 
personnel  to  use  in  assessing  the  risk  to  the  Air  Force  of  being  able 
to  provide  adequate  support  for  mission-critical  software. 

b.  It  is  important  for  AFOTEC  personnel  to  understand  and 

properly  apply  the  RAMSS  for  best  results.  Because  of  the  natural 
transition  of  personnel  it  is  difficult  for  AFOTEC  to  maintain 

corporate  knowledge.  It  is  strongly  recommended  that  a  RAMSS  system 
manager  position  be  created  and  filled  by  a  person  who  can  provide 
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long-term  continuity  for  the  methodology,  automated  tool  support,  and 
implementation  of  software  supportabi 1 ity  evaluation  guidelines. 

c.  The  data  collection  for  evaluation  data  and  maintenance 
release  data  should  be  continued.  New  data  should  be  entered  into 
the  RAMSS  historical  data  bases  and  the  resulting  RAMSS  software 
supportabi 1 ity  risk  regression  equations  updated.  The  RAMSS  can  only 
be  as  effective  as  the  accuracy  and  currency  of  its  baseline  histori¬ 
cal  data. 
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Overview  Briefing  of  RAMSS 
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APPENDIX  A 

OVERVIEW  BRIEFING  OF  RAMSS 

The  purpose  of  this  appendix  is  to  provide  AFOTEC  with  a  set 
of  materials  which  can  be  used  to  brief  the  background,  purpose  and 
procedures  of  the  risk  assessment  methodology  for  software  support- 
ability  (RAMSS).  The  materials  are  presented  in  a  storyboard  fashion 
to  permit  a  briefer  to  understand  the  information  contained  in  each 
slide.  The  materials  presented  here  are  probably  not  suited  to  every 
situation.  Therefore,  the  briefer  may  need  to  tailor  the  materials 
to  varying  purposes  and  audiences.  In  any  event,  the  materials  con¬ 
tained  in  the  appendix  will  provide  a  place  to  start  when  require¬ 
ments  for  general  information  on  the  RAMSS  exist. 
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APPENDIX  B 

RISK  ASSESSMENT  METHODOLOGY  FOR  SOFTWARE  SUPPORTABILITY 
(RAMSS):  EVALUATOR'S  GUIDE 

a.  The  purpose  of  this  appendix  is  to  provide  the  Software  Test 
Manager  (STM)  and  Deputy  for  Software  Evaluation  (DSE)  with  the 
information  needed  to  accomplish  the  Air  Force  Operational  Test  and 
Evaluation  Center's  (AFOTEC's)  software  supportabi 1 ity  risk  assess¬ 
ment.  The  accumulation  of  procedures,  analysis,  and  methodology  is 
denoted  as  the  Risk  Assessment  Methodology  for  Software  Support- 
ability  (RAMSS). 

b.  This  appendix  is  an  evolutionary  document  that  should  be 
updated  periodically.  The  form  of  the  risk  assessment  is  dependent 
upon  the  current  AFOTEC  software  supportabi 1 ity  evaluations,  the 
historical  data  base  of  software  supportabi 1 i ty  evaluations,  and  the 
historical  data  base  of  software  maintenance  release  data. 

c.  This  appendix  is  intended  to  be  a  volume  in  a  series  of  Soft¬ 
ware  Operational  Test  and  Evaluation  Guidelines  prepared  by  the  Soft¬ 
ware  Evaluation  Division  of  the  Logistics  Directorate.  It  is 
intended  for  use  in  the  operational  test  and  evaluation  of  software. 
Comments  should  be  directed  to  the  Office  of  Primary  Responsibility 
(OPR).  The  series  of  guidelines  are: 

(1)  AFOTEC  Pamphlet  800-2,  Volume  1--Management  of  Software 
Operational  Test  and  Evaluation 

(2)  AFOTEC  Pamphlet  800-2,  Volume  2--Reserved 

(3)  AFOTEC  Pamphlet  S00-2,  Volume  3--Software  Maintainabil¬ 
ity  -  Evaluator's  Guide 
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(4)  AFOTEC  Pamphlet  800-2,  Volume  4— Software  Operator- 

Machine  Interface  -  Evaluator's  Guide 

(5)  AFOTEC  Pamphlet  800-2,  Volume  5— Software  Support  Facil¬ 
ity  Evaluation  -  User's  Guide 

(6)  AFOTEC  Pamphlet  800-2,  Volume  6— Reserved. 

d.  Additional  documents  required  to  understand  the  RAMSS  and  its 
application  Include: 

(7)  RAMSS  User's  Handbook,  BDM/ABQ-85-1270-TR 

(8)  RAMSS  Software  Life  Cycle  Evaluator's  Guide, 
B0M/ABQ-86-0090-TR ,  Appendix  A 

This  Guide  Is  organized  as  follows: 
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B.l  GENERAL. 


a.  Software  supportabillty  Is  a  measure  of  the  adequacy  of  per¬ 
sonnel,  resources,  and  procedures  to  facilitate  the  support 
activities  of  modifying  and  installing  software,  establishing  an 
operational  software  baseline,  and  meeting  user  requirements.  Soft¬ 
ware  supportabillty  is  a  function  of  the  quality  of  the  software 
products,  the  capabilities  of  the  software  support  resources,  and  the 
life  cycle  management  processes  which  control  the  procurement,  devel¬ 
opment,  operation,  and  support  of  the  software. 

b.  The  software  supportabillty  risk  is  the  likelihood  that  the 
Air  Force  supporting  command  will  not  be  able  to  accomplish  the 
necessary  support  of  the  software  with  planned  or  actual  support 
resources. 


c.  The  focus  of  this  guide  is  upon  the  process  which  the 
responsible  evaluator  should  apply  in  order  to  derive  the  software 
supportabillty  risk. 

B.2  OVERVIEW  OF  METHODOLOGY:  RESPONSIBILITY,  USE,  RESULTS. 


a.  The  RAMSS  evaluator  will  usually  be  the  STM  and/or  the  DSE. 
The  STM/DSE  should  read  paragraphs  B.l  through  B.9  in  their  entirety 
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and  undtrstand  the  RAMSS  concept  and  procedures  before  beginning  any 
risk  assessment.  These  pages  provide  the  evaluator  with: 

(1)  A  background  of  the  RAMSS  development 

(2)  A  basic  understanding  of  the  RAMSS  procedures 

(3)  Detailed  instruction  for  use  of  the  RAMSS  automated  tool 
support  capabilities  for  analysis  and  reporting  require¬ 
ments. 

b.  The  RAMSS  uses  the  results  from  the  AFOTEC  software  support- 
ability  evaluations  of  the  software  life  cycle  process,  software  product 
maintainability,  and  software  support  resources,  along  with  historical 
software  evaluation  and  maintenance  release  data,  to  determine  the 
software  supportabi 1 ity  risk.  In  addition,  analysis  reports  enable 
the  evaluator  to  determine  which  supportabi 1 ity  factors  are  rated  low 
relative  to  the  historical  evaluation  data,  and  which  supportabi 1 i ty 
factors  have  the  most  impact  on  the  software  supportabi! ity  risk. 
Guidelines  are  presented  to  enable  the  evaluator  to  classify  the 
software  supportabi 1 ity  risk  as  high,  medium,  or  low.  The  high-level 
flow  of  the  RAMSS  is  illustrated  in  figure  B-l.  The  software  sup- 
portability  evaluation  hierarchy  is  shown  in  figure  B-2  to  the  level 
required  by  RAMSS. 

c.  All  required  input  data  and  output  analysis  reports  for  RAMSS 
are  managed  by  the  RAMSS  automated  support  system  described  in  the 
RAMSS  User's  Handbook.  A  functional  flow  of  the  automated  support 
system  is  shown  in  figure  B-3.  The  RAMSS  automated  support  system  is 
menu-driven  and  uses  IBM-PC/AT  or  compatible  hardware,  a  dBase  III 
data  base  management  system,  and  BMDP  statistical  software.  The 
basic  functions  of  the  automated  support  system  for  RAMSS  include: 
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Figure  B-2.  Elements  of  Software  Supportabi 1 ity  Evaluations 
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(1)  Entry/update/report  of  evolving  user/supporter  baseline 
workload  estimate  of  software  support  resources,  block 
release  change  profiles,  and  estimated  risk 

(2)  Entry/update/report  of  software  supportabi 1 Ity  evaluation 
results  and  evaluated  risk 

(3)  Output  of  various  dBase  III  analysis  reports 

a)  Report  of  software  supportabi! ity  evaluation  showing 
percentile  of  evaluation  ratings  relative  to  all 
other  systems  in  the  historical  data  base 

b)  Report  of  software  supportabi 1 Ity  evaluation  risk 
reduction  drivers 

c)  Report  plot  of  workload  in  person  months  per  change 
versus  risk 

d)  Summary  report  of  Important  risk  assessment  results 

(4)  Entry /update/report  of  historical  evaluation  and  main¬ 
tenance  release  data 

a)  dBase  III  data  base  reports 

b)  BHOP  statistical  analysis  reports. 

d.  The  RAMSS  automated  support  system  Interfaces  are  through 
console  menu  selection  and  data  entry,  and  output  reports  generated 
on  the  printer.  The  system  is  very  simple.  The  system  does  not 
provide  a  wide  variety  of  automated  "what  If”  analysis  or  custom 
reports.  Its  focus  is  upon  providing  a  basic  capability  to  enter 
evaluation  data,  receive  an  assessment  of  the  associated  software's 
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supportability  risk  through  printed  reports,  and  update  the  necessary 
historical  data  bases. 

e.  Detailed  requirements  for  use  of  the  RAMSS  automated  support 
system  are  described  in  the  RAMSS  User's  Handbook. 

B.3  PROCEDURE  FOR  APPLYING  RAMSS  TO  AFOTEC  PROGRAMS. 

a.  Application  of  RAMSS  to  AFOTEC  programs  is  appropriate  when¬ 
ever  the  system  contains  significant  or  critical  software  systems  for 
which  Air  Force  software  support  during  post-deployment  support  of 
the  system  is  required. 


1 


$ 


b.  Risk  assessment  of  software  supportability  is  a  life  cycle 
process.  There  are  key  points  (such  as  milestones  0,  I,  2,  3, 
critical  design  review,  IOC,  PMRT)  throughout  a  software  system's 
life  cycle  where  application  of  a  RAMSS  (or  some  part  of  It)  would  be 
beneficial.  Benefits  which  might  occur  Include:  early  planning  and 
trade-off  studies  for  software  support  resource  requirements;  early 
view  of  potential  software  support  management  problems;  early 
visibility  of  user  requirements  for  expected  software  support 
actions;  capability  to  trace  software  supportability  risk  profile 
(i.e.,  measures  of  risk)  throughout  the  life  cycle;  early  view  of 
expected  software  supportability  risk  drivers;  and  the  actual  assess¬ 
ment  of  the  risk  to  user  and  supporter  which  must  be  accepted  before 
support  of  the  software  can  be  assumed. 


3 

8 

t 


«v, 


c.  The  general  RAMSS  procedure  is  illustrated  in  figure  B-4.  The 
application  of  RAMSS  throughout  the  software  life  cycle  process  as 
Integrated  with  AFOTEC  OT&E  phases  and  functions  is  shown  in 
figure  B-5.  This  chart  illustrates  the  areas  of  emphasis  for  AFOTEC 
involvement  using  the  RAMSS  from  early  concept  exploration  through 
post-deployment  support.  These  areas  of  emphasis  reflect  the  tai¬ 
loring  of  the  software  supportability  evaluations  from  which  results 
will  be  input  to  the  RAMSS. 
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PLAN  EVALUATION 

• 
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Figure  8-4. 


Integration  of  RAMSS  and  the  Software 
Supportabi 1 ity  Evaluation  Process 
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d.  Specific  evaluator  guidance  for  evaluation  of  a  user/supporter 
baseline  estimate.  Integration  of  software  supportabll ity  evaluation 
results,  generation  of  analysis  reports,  and  reporting  the  risk 
assessment  results  Is  contained  In  the  next  sections. 

B.4  DERIVING  A  USER/SUPPORTER  BASELINE  ESTIMATE. 

a.  The  user/supporter  baseline  estimate  is  simply  an  estimation 
of  the  support  resources  and  software  change  activity  expected  for  a 
given  software  system  for  one  or  more  block  releases  during  post¬ 
deployment  software  support.  This  estimate  is  derived  by  reviewing 
historical  software  maintenance  data,  available  acquisition  planning 
information  in  documents  such  as  the  CRISP  or  0/S  CMP,  the  current 
software  system  status,  and  the  perspective  of  the  using  and 
supporting  command  personnel. 

b.  The  process  of  deriving  a  baseline  estimate  may  iterate  until 
a  reasonable  consensus  or  compromise  is  reached  among  the  using  and 
supporting  command  personnel,  and  AFOTEC  STM/DSE  personnel.  The 
basic  four  steps,  which  may  be  repeated,  are  shown  in  figure  8-6. 


STEP  1 :  OERIVE  DRAFT  OF  ESTIMATE  USING  THE  RAMSS  AUTOMATED 
SUPPORT  SYSTEM 

STEP  2:  OBTAIN  REVIEW  COMMENTS  ON  THE  DRAFT  FROM  USING 
COMMAND  PERSONNEL 

STEP  3:  OBTAIN  REVIEW  COMMENTS  ON  THE  DRAFT  FROM 
SUPPORTING  COMMAND  PERSONNEL 

STEP  4:  WORK  OUT  COMPROMISE  BETWEEN  USING  AND  SUPPORTING 
COMMAND  ON  DRAFT  AND  UPDATE  NEXT  DRAFT  ON  RAMSS 
AUTOMATED  SUPPORT  SYSTEM 


Figure  B-6.  User/Supporter  Baseline  Estimate  Evolution  Steps 
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c.  The  inputs  for  the  baseline  estimate  are: 

(1)  System  description  names 

(2)  Support  resources  in  form  of  release  schedule  and  support 
personnel  (full-time  equivalents  and  skill  level) 

(3)  Block  release  change  profile  (number,  type,  complexity, 
priority)  for  up  to  three  blocks  (manual  input  and  input 
from  maintenance  release  data  or  other  baseline  estimates 
is  possible). 

d.  The  outputs  of  the  baseline  estimate  computations  for  each 
block  release  are: 

(1)  Available  person  months  per  change 

(2)  Estimated  (optimum)  person  months  per  change 

(3)  Estimated  software  supportabi 1 i ty  risk  based  upon  the 
baseline  estimate  workload  parameters. 

Threshold  and  goal  values  of  0.50  and  0.20  are  reasonable  boundaries 
for  defining  high,  medium,  and  low  risk. 

e.  An  example  report  of  a  user/supporter  baseline  estimate  is 
shown  in  figure  8-7.  Details  for  use  of  the  RAMSS  automated  support 
system  can  be  found  in  the  RAMSS  User's  Handbook. 

8.5  INTEGRATING  SOFTWARE  SUPPORTABILITY  EVALUATION  RESULTS. 

a.  The  steps  to  integrating  the  software  supportabi 1 ity  evalua¬ 
tion  scores  in  order  to  derive  the  evaluated  software  supportabi 1 i ty 
risk  assessment  results  are: 
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(1)  Step  1:  Obtain  the  following  34  evaluation  characteris¬ 
tics  scores  from  the  software  supportabi 1 ity  evaluations: 

a)  Documentation:  Modularity,  Descriptiveness,  Consis¬ 
tency,  Simplicity,  Expandability,  Instrumentation 

b)  Source  Listings:  Modularity,  Descriptiveness,  Con¬ 
sistency,  Simplicity,  Expandability,  Instrumentation 

c)  Personnel:  Management,  Technical,  Support, 

Contractor 

d)  Support  Systems:  Host,  Software  Bench,  Laboratory 

Integrated  Test,  Operational  Integrated  Test,  Con¬ 
figuration  Management  System,  Other 

e)  Facilities:  General  Office  Space,  Support  Systems 

Environment 

f)  Project  Management:  Planning,  Organization  Struc¬ 

ture,  Design  Methods,  Code/Implementation  Methods, 
Test  Strategies,  Project  Interfaces 

g)  Configuration  Management:  Identification,  Control, 

Status  Accounting,  Audit/Review. 

(2)  Step  2:  In  addition  to  the  34  evaluation  scores  of 

step  1,  entry  is  required  of  an  important  overall 

assessment  score  which  is  called  the  software  supporta- 
bility  confidence.  On  the  basis  of  all  available  evalua¬ 
tion  data,  software  system  review  information,  working 
group  data,  and  so  forth,  the  software  test  manager/ 
deputy  for  software  evaluation  assesses  the  confidence 
that  the  subject  software  system  can  be  supported  at  the 
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level  of  activity  indicated  by  the  user/supporter  base¬ 
line  estimate.  This  is  a  value  between  0  (low)  and  1 
(high).  It  is  only  used  as  part  of  the  future  risk 
regression  equation  update  process.  See  the  RAMSS  User's 
Handbook  for  further  information  on  the  update  process. 
The  confidence  value  does  not  affect  results  of  the 
current  evaluation. 

(3)  Step  3:  Enter  the  evaluation  scores  into  the  RAMSS  data 
base.  The  user  enters  the  34  evaluation  scores  plus  the 
confidence  assessment  score  into  the  RAMSS  evaluation 
data  base.  If  desired,  the  low-level  software  life  cycle 
process  evaluation  scores  (see  RAMSS  Software  Life  Cycle 
Evaluator's  Guide  and  RAMSS  User's  Handbook)  can  be 
entered  instead  of  the  ten  level  3  characteristic  scores. 
The  screen  input  format  is  illustrated  in  figure  B-8. 


RAMSS  03/14/86  SCREEN  1.3. 1.1 

SYSTEM:  SWSYSTEM:  SWTYPE:  C-E  SWSYSID: 

RAMSS  SOFTWARE  SUPPORTIBILITY  EVALUATION  SCORES 


♦  LIFE  CYCLE  PROCESS  3.32 
PROJECT  MANAGEMENT  3.47 
Planning 

Organizational  Structure 
Design  Methods 
Implementation  Methods 
Test  Strategies 
Project  Interface 


‘PRODUCT  4.15 

DOCUMENTATION  3.97 
3.33  Modularity 
3.33  Descriptiveness 
4.00  Consistency 
3.50  Simplicity 
3.87  Expandability 
3.00  Instrumentation 


♦SUPPORT  RESOURCES  3.94 
PERSONNEL  3.53 


CONFIGURATION  MANAGEMENT  3.17  SOURCE  LISTINGS  4 
Identification  3.33  Modularity 

Configuration  Control  3.50  Descriptiveness 

Status  Accounting  3.00  Consistency 

Audit  2.83  Simplicity 

Expandabi 1 ity 

**  Computed  Overall  Score  3.80  Instrumentation 

«*  Evaluated  Risk  0.55 

S/W  Supportabillty  Confidence  Assessment 
ENTER  OPTION  (E-EDIT;  S-SAVE;  W-WHAT  IF;  R-RETURN; 
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Figure  8-8.  RAMSS  Screen  Entry  of  Software 

Supportabi 1 ity  Evaluation  Results 


s 

s 


v.'.w.v. 


i 

R 

IS 


'■>  V.vv 


i. v  i ,*  (.4  |ai'f .(  Ki  |.|  'i 


THE  BDM  CORPORATION 


BDM/ABQ-86-0360-TR 


(4)  Step  4:  Compute  necessary  hierarchical  evaluation  scores 
and  associated  risk  values.  This  step  is  conducted 
partially  when  the  evaluation  data  is  entered,  and 
partially  when  RAMSS  printed  reports  are  generated.  This 
step  does  not  require  any  direct  evaluator  participation 
other  than  generating  the  reports  through  menu  selection. 

B.6  OBTAINING  RISK  ASSESSMENT  RESULTS. 

The  software  supportabil ity  risk  assessment  results  are 
contained  in  six  dBase  1 1 T  reports  and  seven  BMDP  reports  which  can 
be  generated  through  the  RAMSS  automated  support  system.  Each  of 
these  reports  is  briefly  described  in  the  following  paragraphs. 
Examples  of  the  dBase  III  analysis  reports  are  illustrated.  The  BMDP 
example  reports  and  further  interpretation  of  all  example  reports  are 
in  the  RAMSS  User's  Handbook. 

B.6.1  dBase  III  Risk  Assessment  Analysis  Reports.  There  are  six 
possible  dBase  III  reports  which  contain  risk  assessment  results.  In 
addition,  there  are  five  raw  data  reports  which  are  essentially 
formatted  reports  of  all  the  data  in  the  historical  evaluation  and 
maintenance  release  data  bases,  and  various  analysis  parameters 
derived  from  the  data  bases. 

B.6.I.1  Report  Al:  User/Supporter  Baseline  Estimate.  This  report 
(see  example  in  figure  B-7)  contains  the  baseline  estimate  inputs  as 
well  as  the  computed  available  person  months  per  change,  estimated 
(optimal)  person  months  per  change,  and  the  estimated  software 
supportabi  1  ity  risk  for  each  of  up  to  three  block  releases.  This 
report  is  used  as  an  input  to  the  software  supportabi 1 ity  evalua¬ 
tions,  and  to  perform  trade-off  analysis  for  support  resources 
(personnel,  skill  level,  and  release  cycle)  and  the  baseline  change 
profiles  (number,  type,  complexity,  priority  of  block  release 
changes).  The  estimated  person  months  per  change  is  computed  from  a 
linear  regression  model  using  support  resources  and  baseline  change 
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profile  parameters  (see  section  8.7).  The  estimated  values  can  be 
used  for  early  computer  resources  planning. 

B.6.1.2  Report  A2:  Table  of  Evaluation  Scores.  This  report  (see 
example  in  figure  B-9)  contains  all  input  and  accumulated  software 
supportabil ity  evaluation  scores  for  levels  3,  2,  1,  and  0  of  the 
evaluation  hierarchy.  In  addition,  the  evaluated  software  support- 
ability  risk  is  output.  This  evaluated  risk  is  computed  from  a 
linear  regression  model  using  these  five  factors:  software  life 
cycle  process,  software  product  maintainability,  support  personnel, 
support  systems,  and  support  facilities.  The  evaluated  risk  can  be 
used  to  report  potential  areas  of  deficiency.  If  the  lower  level 
Software  Life  Cycle  Process  evaluation  scores  are  entered,  then 
another  report  page  will  be  output  containing  those  evaluation 
scores. 


B.6.1.3  Report  A3:  Major  Factor  Percentile  Chart.  This  report  (see 
example  in  figure  8-10)  illustrates  in  a  line  graph  the  percentiles 
for  each  of  the  criteria  and  major  factor  evaluation  scores  relative 
to  the  historical  evaluation  data  base.  Scores  above  75  percent  are 
high,  scores  below  25  percent  are  low.  Low  scores  may  reflect 
deficiencies.  The  percentiles  can  be  shown  relative  to  all  systems 
and  relative  to  all  systems  of  the  same  type  as  the  system  being 
evaluated.  The  example  in  figure  B-10  is  relative  to  systems  of  the 
same  type. 

B.6.1.4  Report  A4:  Major  Factor  Risk  Reduction  Chart.  This  report 
(see  example  in  figure  B-ll)  illustrates  in  a  line  graph  the  maximum 
reduction  in  evaluated  risk  possible  for  each  criteria  and  major 
factor.  Those  criteria/major  factors  which  can  effect  large  reduc¬ 
tions  in  evaluated  risk  are  termed  risk  drivers  and  are  prime 
candidates  for  further  analysis  of  potential  risk  reduction. 

8.6. 1.5  Report  A5:  Plot  of  Cumulative  Distribution  of  Person  Months 
Per  Change  Versus  Risk.  This  report  (see  example  in  figure  B - 1 2 )  is 
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a  plot  showing  the  cumulative  distribution  and  a  table  of  evaluated 
and  estimated  risk  and  person  months  per  change.  The  plot  can  be 
used  for  quick  “what  If"  analysis  of  changes  to  the  person  months  per 
change  workload  and/or  the  risk  values.  Plots  are  produced  for  data 
relative  to  each  of  the  three  block  releases  in  the  user/supporter 
baseline  estimate. 

B.6.1.6  Report  A6:  Summary  of  RAMSS  Results.  This  report  (see 
example  In  figure  B-13)  Is  a  compact  summary  of  Information  from  the 
reports  A1  through  A5. 

B.6.1.7  Report  Dl:  Evaluation  Data  Base.  This  report  (see  the 
RAMSS  User's  Handbook  for  examples  of  all  following  reports)  is  a 
formatted  table  of  all  fields  In  the  evaluation  data  base.  This 
report  is  primarily  used  as  a  printed  copy  of  the  evaluation  data. 

B.6.1.8  Report  02:  Maintenance  Release  Data  Base.  This  report  is  a 
formatted  table  of  all  fields  in  the  maintenance  release  data  base. 
This  report  Is  primarily  used  as  a  printed  copy  of  the  evaluation 
data. 


B.6.1.9  Report  D3:  Table  of  Evaluated  Risk  Regression  Equation 
Coefficients.  This  report  lists  all  coefficients  used  in  the 
evaluated  risk  regression  equation  and  the  equations  necessary  to 
compute  the  evaluated  risk. 

B. 6.1. 10  Report  04:  Table  of  Estimated  Person  Months  Per  Change 
Regression  Equation  Coefficients.  This  report  lists  all  the  coeffi¬ 
cients  used  in  the  estimated  person  months  per  change  regression 
equation  and  the  equations  necessary  to  compute  the  estimated  person 
months  per  change. 
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1*.  »» 


i  I't: 


iH'  ‘i’A  ||*  fa*  Ji«  ~ l»; 


THE  BOM  CORPORATION 


8DM/ABQ-86-G360-TR 


B.6.1.11  Report  05:  Table  of  AFOTEC  Parameters  ( Threshold/Goal ). 


This  report  lists  the  threshold  and  goal  values  for  software 
supportablllty  evaluation  scores  and  the  software  supportablllty  risk 
values.  These  threshold  and  goal  values  can  be  set  by  AFOTEC  person¬ 
nel  and  are  only  used  to  determine  whether  an  evaluation  score  and/or 
risk  value  is  considered  to  be  HIGH,  MEDIUM,  or  LOW. 

B.6.2  BMOP  Reports.  There  are  seven  possible  BMDP  reports  which 
contain  detailed  statistical  analysis  data  concerning  the  evaluation 
data,  the  evaluated  risk  regression  model,  the  maintenance  release 
data,  and  the  estimated  risk  regression  model.  Data  Is  passed  to 
BMOP  through  ASCII  files  written  by  dBase  III  copy  commands. 

B.6.2. 1  Bl:  Simple  Data  Description.  This  report  lists  all  Input 
evaluation  data,  various  anomaly  checks  of  the  data,  and  univariate 
statistics. 

B.6.2. 2  B2:  Histogram  and  Univariate  Plots.  This  report  provides  a 
histogram  and  cumulative  distribution  plot  of  each  major  factor 
evaluation  score. 

B.6.2. 3  B3:  Multiple  Linear  Regression.  This  report  provides  the 
coefficients  for  the  evaluated  risk  regression  model.  These  coeffi¬ 
cients  must  be  manually  entered  Into  a  dBase  III  file  each  time  new 
or  modified  evaluation  data  Is  Included  in  the  regression  analysis. 

B.6.2. 4  B4:  Simple  Data  Description.  This  report  Is  similar  to  Bl, 
except  It  Is  for  maintenance  release  data. 

B.6.2. 5  85:  Histogram  and  Univariate  Plots.  This  report  Is  similar 
to  B2,  except  It  Is  for  maintenance  release  data. 

B.6.2. 6  B6:  Description  of  Groups.  This  report  provides  histograms 


and  analysis  of  variance  Information  on  certain  stratified  groups  of 
maintenance  release  data  variables. 


I  «.«  a.I  I 
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B.6.2.7  B7:  Multiple  Linear  Regression.  This  report  is  similar  to 
B3,  except  It  Is  for  maintenance  release  data  and  the  estimated 
person  months  per  change.  The  standard  error  estimate  as  well  as  the 
regression  coefficients  must  be  manually  entered  Into  a  dBase  III 
file  each  time  new  or  modified  maintenance  release  data  are  Included 
In  the  regression  analysis. 

B.7  ANALYZING  RISK  ASSESSMENT  RESULTS. 

The  categorization  of  the  risk  assessment  results  as  HIGH, 
MEDIUM,  or  LOW  depends  upon  the  values  which  distinguish  the  thresh¬ 
old  and  goal  risk.  The  computation  of  the  risk  values  depends  upon 
the  linear  regression  models.  The  analysis  Is  primarily  aided  by  the 
dBase  III  and  BMOP  printed  reports. 

B.7.1  Settlnq/Using  Threshold  and  Goal  Values 


a.  The  recommended  values  for  threshold  and  goal  are: 


(1)  Software  Supportabil ity  Evaluation 
Goal:  5.0 

Threshold:  3.5 

(2)  Software  Supportabil Ity  Risk 
Goal:  0.20 

Threshold:  0.5 

(3)  Software  Supportabil Ity  Percentiles 
Goal:  75X 

Threshold:  25% 

These  values  are  based  upon  experience  and  the  current  hlstorlal  data 
base.  They  are  somewhat  subjective,  and  need  to  evolve  over  time. 
The  values  are  used  only  as  a  reference  In  the  summary  report  so  as 
to  distinguish  scores  which  are  HIGH,  MEDIUM,  or  LOW. 
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b.  Th«  current  threshold  and  goal  values  are  set  through  a 
dBASE  III  screen  If  they  need  to  be  modified. 

B.7.2  The  Estimated  Risk  Regression  Equation. 

a.  The  estimated  risk  Is  determined  from  user/supporter  baseline 
estimate  parameters.  First,  a  regression  equation  Is  used  to  deter¬ 
mine  the  estimated  workload  In  person  months  per  change.  Next,  the 
available  person  months  per  change  Is  computed  from  user/supporter 
baseline  estimate  parameters.  Finally,  the  estimated  risk  is 
determined  using  a  normal  distribution  of  regression  equation 
residuals  with  mean,  the  estimated  person  months  per  change,  and 
standard  deviation,  the  standard  error  of  estimate  of  the  regression 
equation.  The  estimated  risk  Is  the  area  under  this  normal  curve 
above  the  available  person  months  per  change  value.  The  regression 
equation  for  estimated  person  months  is  determined  from  the  histori¬ 
cal  maintenance  release  data.  The  equations  are  as  follows: 

PMPC  =»  person  months  per  change 

PMPC"  *  estimated  person  months  per  change 

PMPC"  =  eL~ 

where 

L"  =  b0  +  bi  (AVGSKILL)  +  bg  (PTCORR)  +  b3  (PCLOW) 

+  b4  (PCHIGH)  +  bs  (PPNORM) 

10 

+  I  bi  (TYPE,) 

13o 

AVGSKILL  -  average  skill  (1-Low  to  5-High)  of  support 
personnel 

PTCORR  *  percentage  of  change  requests  which  are  corrections 

PCLOW  *  percentage  of  change  requests  which  are  low 

complexity 

PCHIGH  *  percentage  of  change  requests  which  are  high 

complexity 
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PPNORM  =  percentage  of  change  requests  which  are  normal 
pr i or i ty 

TYPE i  =  1  if  system  type  is  same  as  TYPE i 
=  0  otherwise 

The  coefficients  bj  are  determined  from  a  BMDP  regression  analysis 
program.  These  coefficients  will  change  whenever  the  historical 
maintenance  release  data  base  is  updated.  Example  coefficients  are 
shown  in  figure  8-11.  The  bo  coefficient  (INTERCEPT)  incorporates 
the  coefficient  for  the  types  SUP  and  SIM,  so  these  types  do  not  have 
coefficients  specifically  specified  in  figure  B-14. 

b.  The  available  person  months  per  change  for  a  block  release  is 
determined  from  the  user/supporter  baseline  estimate  parameters  for 
total  number  of  personnel,  total  percentage  dedicated  to  the  software 
release  (includes  percentage  dedicated  to  the  software  system  and  any 
release  overlap),  duration  of  the  release  cycle,  and  total  number  of 
changes  in  the  release. 

PMPC^  =  available  person  months  per  change 

=  (Number  Persons  *  Percent  Dedicated  *  Release 
Ouration)/Number  Changes 

Estimated  risk  for  an  available  PMPC,  PMPCA,  is  calculated  as 

.  JLn( PMPC. )  -  L~ 

R  =  Estimated  Risk  =  1  -  F  - - - 

where 

F(x)  =  cumulative  standard  normal  distribution  function 

evaluated  at  x 

s  =  an  estimate  of  the  standard  deviation  of  the  normal 

distribution  of  the  available  person  months  per 
change  about  the  L  . 
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REGRESSION  TITLE  IS 

3W  TYPE  DUMMY  VARIABLES  PLUS  COVARIATES 


0  DEPENDENT  VARIABLE . 

TOLERANCE  .  . 

ALL  DATA  CONSIDERED  AS  A  SINGLE  GROUP 
OMULTIPLE  R  0.3774 

MULTIPLE  R-SQUARE  0.3334 


33  LN(PMPC) 

0 . 0 1 00 


STD.  ERROR  OF  EST. 


0. 9509 


ANALYSIS  OF  VARIANCE 

REGRESSION 

RESIDUAL 


SUM  OF  SQUARES 
63 . 3334 
136.3911 


DF 
10 
1  40 


MEAN  SQUARE 
6 . 

0. 9042 


F  RATIO 

7 . 003 


P  (TAIL) 

0. 0000 


BWBBH — BBWWBWWB  BWPWWWW  UJ  W  M  ftWMW  WWWW. 

/s 

s 
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STD.  REG 


VARIABLE 

COEFFICIENT 

STD.  ERROR 

COEFF 

T  P  ( 

2  TAIL) 

TOLERANCE 

INTERCEPT 

0. 07083 

AVGSKILL 

7 

0. 34S37 

0. 13224 

0.  323 

4.  147 

0 • ooo t 

0.  73316 

PTCORR 

29 

-0.02948 

0.39983 

-0. 007 

-0 .074 

0.9413 

0. 32019 

PCLOW 

32 

0.21813 

0 . 40660 

0.  032 

0. 337 

0.3924 

0. 3 1 666 

PCHIGH 

34 

2.09363 

0. 78837 

0.233 

2.658 

0 . 00-38 

0.51531 

PPNORM 

33 

-1.57328 

0.61743 

-0. 208 

-2.346 

0 . 0 1 20 

0.71313 

ATD 

39 

-0. 30843 

0. 49983 

-0 . 08 1 

-1.017 

0.3108 

0. 74367 

ATE 

40 

0. 14413 

0. 43891 

0.  033 

0 . 328 

0.7431 

0. 46024 

C-E 

41 

0 • 33233 

0.29391 

0.  133 

1 . 200 

0.2324 

0.28366 

EW 

42 

0.83802 

0. 46308 

0.171 

1 . 853 

0. 0660 

0. 3365o 

OFP 

43 

0.04114 

0.29273 

0.017 

0.  141 

0.8884 

0. 22299 

Figure  B-14. 

Example  Report: 

Baseline 

Estimate 

Risk  Regression  Analysis 

The  value  of  "s"  Is  obtained  from  the  BMDP-generated  table  of  regres¬ 
sion  results  ("STD.  ERROR  OF  EST."  In  figure  B-14);  the  function  F 
may  be  numerically  approximated  by 


F  (x) 

G(z) 


I- 


SGN(x) 


ir 

2* 


1  -  (1  +  a^z 


2  3  4 

a2z  +  a^z  +  a4z  ) 


-4 


a2  -  0.278393 
a2  -  0.230389 
a3  -  0.000972 
a4  =  0.078108 


SGN(x)  *  1  If  x  >  0 

-1  If  x  <  0 
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c.  As  an  example  from  block  3  of  the  user/supporter  baseline 
estimate  In  figure  8-7 ,  If 

AVGSKILL  *  3.0 
PTCORR  -  0.65 
PC LOW  *  0.65 
PCHIGH  -  0.05 
PPNORM  *  0.65 
TYPEc-E  -  1-0 

then 

L‘  *  0.07083  +  (0.54837) (3.0)  +  (-0.02948) (0.65) 

+  (0.21815) (0.65)  +  (2.09568) (0.05) 

♦  (-1.57228) (0.65)  +  (0.35255) (1) 

*1.2739275 

and 

PMPC"  -  3.57 

The  available  PMPC  from  the  block  3  example  of  figure  B-7  Is  computed 
as: 

PMPCA  -  [ (15*.19  +  9*.90)*(0.667)*9.0 J/20 
*  3.285 

The  estimated  risk  for  block  3  of  the  example  is  therefore 


-  1  -  F(-0. 0889272) 

-  1  -  (0.5  +  (-1) (0. 5G(0. 0628810))) 

-  1  -  (0.5  -  0.5(1  -  ( 1 .0184180) _4) ) 

-  1  -  (0.5  -  0.5(0.0704010)) 

*  0.5  +  0.0352005 

-  0.5352005 

as  shown  in  the  example  of  figu-e  B-7. 
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B.7.3  The  Evaluated  Risk  Regression  Equation 

a.  The  evaluated  risk  Is  determined  from  the  software  support- 
ability  evaluation  scores  and  a  regression  equation  derived  from  the 
historical  evaluation  data.  The  equations  are  as  follows: 

R  *  risk  -  1-ACONFID  0  <  R  <  1 


R-  «  R(1  -  a)  ♦  §  a  -  0.02 


R~ 


1  +  e 


-L-  2 


(1  -  a) 


-1 


L~  »  an 


R- 


bo  +  bi  (APRODUCT)  +  b2  (AEPER)  +  b3  (AESYS) 
+  b4  (AEFAC)  +  bs  (AMAHAGE) 


APR00UCT 

AEPER 

AESYS 

AEFAC 

AMANAGE 


Software  Product  Malntalnabil fty  evaluation  score 
Support  Resources  Personnel  evaluation  score 
Support  Resources  System  evaluation  score 
Support  Resources  Facilities  evaluation  score 
Software  Life  Cycle  Process  evaluation  score 


The  coefficients  b^  are  determined  from  a  BMDP  regression  analysis 
program.  These  coefficients  will  change  whenever  the  historical 
evaluation  data  base  is  updated.  Example  coefficients  are  shown  in 
figure  B-15. 
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PAGE  3  BMDP1R  LIRISL)  VS.  SUPPORTABILITY  FACTORS 
REGRESSION  TITLE  IS 
L<RISK)  VS.  SURP0RTA8ILITY  FACTORS 


0  DEPENDENT  VARIABLE . 

TOLERANCE  . 

ALL  DATA  CONSIDERED  AS  A  SINGLE  GROUP 
OMULTIPLE  R  0.8086 

MULTIFLE  R-SQUARE  0.6539 

ANALYSIS  OF  VARIANCE 


13  LRISK 

0.0X00 


STD.  ERROR  OF  EST. 


0.6239 


SUM  OF  SQUARES  DF 

MEAN  SQUARE 

F 

RATIO 

P (TAIL) 

REGRESSION  33. 

2376  5 

7.6473 

1 9 . 630 

0 . 0000 

RESIDUAL 

20. 

2382  32 

0.3892 

STD.  REG 

VARIABLE 

COEFFICIENT 

STD.  ERROR 

COEFF 

T 

P (2  TAIL) 

TOLERANCE 

INTERCEPT 

4.90401 

APRODUCT 

4 

-0. 29131 

0. 13090 

-0.226 

-2.223 

0.0304 

0.64641 

AEPER 

5 

— 0. 15600 

0.12403 

-O. 136 

-1.233 

0.2141 

0. 3661 3 

AESYS 

6 

-0.23120 

0. 12903 

-0. 181 

-1.946 

0.0370 

0.77162 

AEFAC 

7 

0.04294 

0. 10013 

0.043 

0.429 

0.6699 

0.66939 

AMANAGE 

11 

-0. 66174 

0. 14236 

-0. 309 

-4.642 

0. OOOO 

0. 3553 1 

Figure  B-15.  Example  Report:  Evaluation  Risk 
Regression  Analysis 


b.  As  an  example  from  the  evaluation  scores  illustrated  In 
figure  B-9,  If 

APROOUCT  -  4.15 
AEPER  -  3.53 
AESYS  »  3.72 
AEFAC  *  4.58 
AMANAGE  -  3.32 

then 

L“  »  4.90401  +  (-0. 29131) (4. 15)  +  (-0. 15600) (3.53) 

+  (-0.25120)(3.72)  +  (  0.04294) (4.58) 

+  (-0.66174)(3.32) 

-  0.20962 


1  +  e 


-0.20962  '  T2  (1-02)  1 
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B.7.4  Analyzing  Results  of  the  Assessment  Reports 

a.  There  are  six  risk  assessment  reports  output  from  the  RAMSS 
automated  support  system: 

(1)  Al:  User/Supporter  Baseline  Estimate 

(2)  A2:  Table  of  Evaluation  Scores 

(3)  A3:  Major  Factor  Percentile  Chart 

(4)  A4:  Major  Factor  Risk  Reduction  Chart 

(5)  A5:  Plot  of  Cumulative  Distribution  of  Person  Months  Per 

(6)  A6:  Summary  of  RAMSS  Results 

b.  An  example  of  the  report  Al  Is  shown  In  figure  B-7.  Examples 
of  reports  A2,  A3,  A4,  A5,  and  A6  are  shown  in  figures  B-9  through 
B-13,  respectively.  A  brief  explanation  of  these  reports  Is 
contained  In  section  B.6.  Detailed  explanations  of  these  reports  are 
contained  In  the  RAMSS  User's  Handbook. 


c.  For  the  data  in  the  example  reports  the  following  analysis 
conclusions  hold: 

(1)  From  report  A6:  the  evaluated  software  supportabllity 

risk  Is  HIGH;  the  estimated  software  supportabllity  risk 
Is  MEDIUM;  the  main  risk  driver  Is  software  life  cycle 
process;  support  resources  personnel  had  a  somewhat  low 
evaluation  score  but  there  was  not  much  potential  for 
reduction  of  risk  by  improvements  In  this  characteristic 
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(2)  From  report  A4;  the  amount  of  risk  reduction  Is  greatest 
for  software  configuration  management;  this  major  factor 
Is  the  prime  candidate  for  potential  improvement  to 
reduce  software  supportabllity  risk 

(3)  From  report  A3:  there  is  support  for  the  conclusion  that 
the  overall  evaluation  Is  LOW  relative  to  the  systems  In 
the  evaluation  data  base. 

Suppose  the  evaluator  were  able  to  analyze  the  detailed  evaluation 
results  and  conclude  that  the  overall  software  life  cycle  process 
score  could  be  Improved  from  3.32  to  4.58.  The  corresponding 
reduction  In  evaluated  risk  would  be  from  0.55  to  0.35,  and  the  vari¬ 
ous  analysis  reports  would  reflect  that  overall  improvement  In  the 
software  life  cycle  process  evaluation  results. 

B.8  REPORTING  RISK  ASSESSMENT  RESULTS. 

The  risk  assessment  results  which  should  be  reported  include: 

(1)  Evaluated  software  supportabllity  risk  (report  A2) 

(2)  Estimated  software  supportabllity  risk  (report  Al) 

(3)  Major  risk  drivers  (report  A4) 

(4)  Risk  reduction  potential  (report  A4) 

(5)  Individual  characteristics  anomalies  (all  reports) 

a)  Above  risk  threshold  (0.50) 

b)  Below  percentile  threshold  (0.25) 

c)  Below  evaluation  threshold  (3.50) 

d)  Goal  or  better  characteristics  (0.20,  0.75,  5.0) 
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An  overall  assessment  of  the  software  supportablllty  risk  as  HIGH, 
MEDIUM,  or  LOW  on  the  basis  of  the  matrix  in  figure  B-16  should  be 
reported. 


Figure  B-16.  Software  Supportabi 1 ity  Risk  Assessment  Matrix 
B.9  SUMMARY  OF  THE  RAMSS  PHILOSOPHY. 

a.  The  following  is  a  summary  of  the  general  philosophy  which 
should  guide  the  evaluator  (e.g.,  STM/DSE)  in  conducting  a  RAMSS. 

b.  The  evaluator  should  understand  that  the  RAMSS  is  not  yet 
mature.  It  is  critical  that  the  evaluator  review  all  aspects  of  the 
risk  in  order  to  arrive  at  an  overall  assessment  of  a  software's 
supportabi 1 Ity  risk  to  the  Air  Force. 

c.  The  evaluator  should  always  be  prepared  to  describe  why  the 
evaluated  or  estimated  risk  is  HIGH  or  LOW  by  tracing  to  specific 
criteria,  major  factors,  characteristics,  resource  workload,  or  base¬ 
line  change  profile  for  supporting  data. 


:  *v  ^^niLrxrxr. 
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d.  The  evaluator  should  be  very  familiar  with  the  RAMSS  automated 
support  system,  or  there  should  be  an  AFOTEC  support  person  who  is 
familiar  with  It  and  can  assist  the  evaluator. 

e.  The  historical  data  for  evaluations  and  maintenance  releases 
are  immature.  Care  should  be  exercised  in  relying  too  heavily  on 
these  data.  These  data  need  to  be  Improved  over  time.  Anomalies  in 
results  may  be  because  of  Incomplete  data. 


.V.V.V.V.V 
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ATTACHMENT  B1 
GLOSSARY  OF  TERMS 


B. 1.1  INTRODUCTION. 

a.  The  glossary  of  terms  for  the  RAMSS  has  varied  as  the 
methodology  development  has  progressed.  Refer  to  BDM/A-84-322-TR 
(Final)  dated  September  28,  1984,  for  a  complete  glossary  of  terms 
relating  to  risk  assessment. 

b.  Some  terms  have  more  than  one  description;  when  this  is  the 
case,  the  description  either: 

(1)  Are  significantly  different  between  sources  (though  the 
effective  meaning  may  be  not  much  different) 

(2)  Are  used  differently  (different  users  or  technical 
language) 

(3)  May  be  found  within  the  context  of  a  different  source 

(4)  Have  real  differences  in  meaning. 

Both  DoD  and  non-DoD  (e.g.,  FIPS  PUBs,  NBS  Special  Publications) 
sources  are  used.  The  non-DoD  sources  and  terms  are  not  mandated  for 
our  use,  but  are  rather  included  for  breadth  of  understanding,  for 
those  relevant  terms  commonly  used  with  the  non-DoD  governmental 
and/or  private  sectors. 

c.  The  source  of  each  description  is  indicated  by  a  symbo1  m 
parentheses  before  that  source's  term  description: 


t,«  l»  Ll!kj 


- -  —  «.,._._?fWDPRHJni _ 


microcopv 

NA’uiNAI 
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HU«t»  -.UN^- 


CHARI 
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(SYMBOL!. i) 
Description!. !•.. 
(SYMBOL!. 2) 
Oescriptioni. 2* • • 
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(SYMBOL!. n) 
Description!. n... 
TERM2 


TERMN 


The  symbols  used  and  corresponding  sources,  are: 


(AF0TECP1) 

AFOTECP  800-2,  Volume  I,  10  Nov  82,  "Software  Test 
Manager's  Guide." 

(AF0TECP3) 

AFOTECP  800-2,  Volume  III,  1  Jan  84,  "Software  Main¬ 
tainability  Evaluator's  Guide." 

(AF0TECP5) 

AF0TEC  800-2,  Volume  V,  25  Jul  83,  "Software  Support 
Facility  Evaluation--User's  Guide." 

( AFR55-43) 

Air  Force  Regulation  55-43,  "Management  of  Opera¬ 
tional  Test  and  Evaluation",  28  Jun  1985. 

(AFR800-14) 

Air  Force  Regulation  800-14,  Volume  I,  "Management  of 
Computer  Resources  in  Systems,"  12  Sep  75. 

(DoD480A) 

DoD  Standard  480A,  "Configuration  Control  -  Engi¬ 
neering  Changes,  Deviations  and  Waivers",  12  Apr  78. 

(ROWE) 

Rowe,  William,  An  Anatomy  of  Risk,  John  Wiley,  1977. 

(CURRENT) 

Current  document  definition. 
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B. 1.2  GLOSSARY  OF  TERMS  FOR  DEVELOPING  AND  IMPLEMENTING  A  RISK 
ASSESSMENT  METHODOLOGY  FOR  SOFTWARE  SUPPORTABILITY. 

Allocated  8ase1ine 

(DO0480A) 

See  Baseline. 


Allocated  Configuration  Identification 
(DoD480A) 

Current,  approved  performance  oriented  specifications  governing 
the  development  of  configuration  items  that  are  part  of  a  higher 
level  Cl,  in  which  each  specification  (1)  defines  the  functional 
characteristics  that  are  allocated  from  those  of  the  higher  level 
Cl,  (2)  establishes  the'  tests  required  to  demonstrate  achievement 
of  its  allocated  functional  characteristics,  (3)  delineates  neces¬ 
sary  interface  requirements  with  other  associated  configuration 
items,  and  (4)  establishes  design  constraints,  if  any,  such  as 
component  standardization,  use  of  inventory  items,  and  integrated 
logistic  support  requirements. 

Application  Software 

(AF0TECP5) 

The  software  written  by  software  support  personnel,  or  purchased 
from  a  contractor,  used  directly  in  supporting  ECSs.  It  is 
normally  used  for  simulation,  testing,  and  ECS  code  development. 

Automated  Software  Development  Tool 

(AF0TECP5) 

A  component  of  System  Software  that  assists  in  the  design,  imple¬ 
mentation,  documentation,  and  verification  of  ECS  software. 


§ 


Avai labi 1 ity 
(AFR800-14) 

A  measure  of  the  degree  to  which  an  item  is  in  the  operable  and 
commitable  state  at  the  start  of  the  mission,  when  the  mission  is 
called  for  at  an  unknown  (random)  point  in  time.  (MIL -STD-721 ) 

(AF0TECP5) 

The  probability  that  a  system  is  operating  satisf actor i ly  at  any 
point  in  time  when  used  under  stated  conditions. 

Available  Person  Time  (APT) 

(CURRENT) 

The  software  support  person-months  available  for  a  particular 
software  release  computed  as  the  product  of  the  release  duration 
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In  months,  the  number  of  support  personnel,  and  the  percentage  of 
the  time  those  personnel  are  dedicated  to  the  subject  software 
release  (versus  shared  across  other  releases  or  other  software 
systems).  This  time  includes  overhead  activity  directly  related 
to  the  subject  release.  The  release  duration  is  the  release 
engineering  completion  date  minus  the  release  start  date. 

Baseline 

(DoD480A) 

A  configuration  identification  document  or  a  set  of  such  documents 
formally  designated  and  fixed  at  a  specific  time  during  a  Cl's 
life  cycle.  Baselines,  plus  approved  changes  from  those  base¬ 
lines,  constitute  the  current  configuration  identification.  For 
configuration  management  there  are  three  baselines,  as  follows: 

a)  Functional  Baseline.  The  initial  approved  functional  con¬ 

figuration  ident if ication. 

b)  Allocated  Baseline.  The  initial  approved  allocated  con¬ 

figuration  identification. 

c)  Product  Baseline.  The  initial  approved  or  conditionally 
approved  product  configuration  identification. 


(ROWE) 

A  known  reference  used  as  a  guide  for  further  development 
activities. 

Baseline  Profile 

(CURRENT) 

See  Baseline  Software  Change  Profile. 

Baseline  Software  Change  Profile 
(CURRENT) 

The  set  of  numbers  (or  any  subset)  determined  by  specifying  the 
number  of  requests  per  release  for  each  request  category.  A 
request  category  is  the  triple  (type,  priority,  complexity)  where 
type  is  conversion,  enhancement,  or  correction;  priority  is 
emergency,  urgent,  or  normal;  and  complexity  is  high,  medium,  low. 

Baseline  Software  Supportabi 1 ity  Estimate 

(CURRENT) 

See  User/Supporter  Baseline  Estimate 
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Block  Release 

(CURRENT) 

See  Release. 

Change  Control 

(DoD480A) 

See  Configuration  Control 
Complexity  of  MA 
(CURRENT) 

See  Maintenance  Complexity 
Computer  Program 
(AFR800-14) 

A  series  of  instructions  or  statements  in  a  form  acceptable  to  an 
electronic  computer,  designed  to  cause  the  computer  to  execute  an 
operation  or  operations. 

Computer  Program  Configuration  Item  ( CPC  I ) 

(CURRENT) 

See  Computer  Software  Conf iguration  Item 
Computer  Resources 
(CURRENT) 

The  totality  of  computer  hardware,  computer  software,  personnel, 
documentation,  supplies,  and  services. 

( AFR800-I4) 

The  totality  of  computer  equipment,  computer  programs,  associated 
documentation,  contractual  services,  personnel  and  supplies. 

Computer  Resources  Integrated  Support  Plan  (CRISP) 

(AFR55-33) 

The  CRISP  identifies  organizational  relationships  and  responsi¬ 
bilities  for  the  management  and  technical  support  of  computer 
resources.  It  functions  during  the  full-scale  development  (FSD) 
phase  to  identify  computer  resources  necessary  to  support  computer 
programs  after  program  management  responsibility  and  system  turn¬ 
over  are  transferred.  After  the  transfer,  the  CRISP  continues  to 
function  as  the  basic  agreement  between  the  supporting  and  using 
commands  for  management  and  support  of  computer  resources. 


Mb 
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Computer  Resources  Working  Group  (CRWG) 

(CURRENT) 

A  group  comprised  of  all  the  participating  commands  (for  a 
particular  system)  which  writes  and  updates  the  Computer  Resources 
Integrated  Support  Plan  (CRISP).  The  group  insures  that  necessary 
elements  of  the  CRISP  are  included  in  transfer  and  turnover 
agreements. 

Computer  Software  Configuration  Item  (CSCI) 

(CURRENT) 

See  Configuration  Item 
Configuration  Audit 
(CURRENT) 

The  process  of  verifying  that  all  required  conf iguration  items 
have  been  produced,  that  the  current  version  agrees  with  specified 
requirements,  that  the  technical  documentation  completely  and 
accurately  describes  the  configuration  items,  and  that  all  change 
requests  have  been  resolved. 

Configuration  Control 

(Do0480A) 

The  systematic  evaluation,  coordination,  approval  or  disapproval, 
and  implementation  of  all  approved  changes  in  the  configuration  of 
a  configuration  item  after  formal  establishment  of  its  configura¬ 
tion  identification. 

Configuration  Identification 

(DoD480A) 

The  current  approved  or  conditionally  approved  technical  docu¬ 
mentation  for  a  configuration  item  as  set  forth  in  specifications, 
drawings  and  associated  lists,  and  documents  referenced  therein. 

Configuration  Index 

(CURRENT) 

This  document,  produced  by  the  development  contractor,  reports  the 
current  status  of  configuration  item  development  in  terms  of 
specifications  and  other  documents  that  depend  on  the  configura¬ 
tion,  such  as  qualification  Test  Plans  and  Procedures,  User 
Manuals,  and  the  Version  Description  Document.  It  lists  all  ECPs 
and  SCNs  incorporated ,  approved  ECPs  not  yet  incorporated,  and 
other  data. 
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Configuration  Item  (Cl) 

(AFR800-14) 

An  aggregation  of  equipment/software,  or  any  of  its  discrete 
portions,  which  satisfies  an  end  use  function  and  is  designated  by 
the  Government  for  configuration  management.  CIs  may  vary  widely 
in  complexity,  size  and  type,  from  an  aircraft  or  electronic 
system  to  a  test  meter  or  round  of  ammunition.  Ouring  development 
and  initial  production,  CIs  are  only  those  specification  items 
that  are  referenced  directly  in  a  contract  (or  an  equivalent 
in-house  agreement).  During  the  operation  and  maintenance  period, 
any  repairable  item  designated  for  separate  procurement  is  a 
configuration  item  (AFR  65-3). 

Configuration  Management  (CM) 

(0oD480A) 

A  discipline  applying  technical  and  administrative  direction  and 
surveillance  to  (1)  identify  and  document  the  functional  and 
physical  characteristics  of  a  configuration  item,  (2)  control 
changes  to  those  characteristics,  and  (3)  record  and  report  change 
processing  and  implementation  status. 

Configuration  Management  Plan  (CMP) 

(CURRENT) 

A  document  which  describes  project  responsibilities  and  procedures 
for  implementing  CM. 

Configuration  Management  System  (CMS) 

(AF0TECP5) 

A  system  applying  technical  and  administrati ve  direction  and 
surveillance  to  identify  and  document  the  functional  and  physical 
characteristics  of  a  configuration  item;  to  control  changes  to 
those  characteristics  and  to  record  and  report  change  processing 
and  implementation  status. 

Configuration  Status  Accounting 

(DO0480A) 

The  recording  and  reporting  of  the  information  that  is  needed  to 
manage  a  configuration  effectively,  including  a  listing  of  the 
approved  configuration  identification,  the  status  of  proposed 
changes  to  the  configuration,  and  the  implementation  status  of 
approved  changes. 

Consistency 

(CURRENT) 

A  measure  of  the  extent  the  software  products  correlate  and 
contain  uniform  notation,  terminology,  and  symbology. 
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Conversion  (Adaptive)  MA 
(CURRENT) 

See  Maintenance  Type. 

Corrective  MA 
(CURRENT) 

See  Maintenance  Type. 

Critical  Issues 
(AF0TECP1 ) 

Those  aspects  of  a  system's  capability,  either  operational,  tech¬ 
nical,  or  other,  that  must  be  questioned  before  a  system's  overall 
worth  can  be  estimated  and  that  are  of  primary  importance  to  the 
decision  authority  in  reaching  a  decision  to  allow  the  system  to 
advance  into  the  next  acquisition  phase  (DoD  Directive  5000.3). 

Data  Item  Description 

( AFR800-14) 

A  form  which  specifies  an  item  of  data  required  to  be  furnished  by 
a  contractor.  This  form  specifically  defines  the  content,  prepa¬ 
ration  instructions,  format  and  intended  use  of  each  data  product. 

Oescriptiveness 

(CURRENT) 

A  measure  of  the  extent  that  software  products  contain  information 
regarding  its  objectives,  assumptions,  inputs,  processing,  out¬ 
puts,  components,  revision  status,  etc. 

Development  Contractor  Activity 

(CURRENT) 

Those  organizations  responsible  for  development  of  a  system  in 
order  to  achieve  an  initial  operational  capability.  Organizations 
include  the  prime  development  contractor  and  any  subcontractors  to 
the  prime  contractor. 

Documentation 

(AF0TECP5) 

AH  of  the  written  work  describing  operating  and  maintenance 
procedures  for  a  system. 

Embedded  Computer  Resources 

(AF0TECP1) 

Computer  resources  incorporated  as  integral  parts  of,  dedicated 
to,  required  for  direct  support  of,  or  for  the  upgrading  or 
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modification  of  major  or  less  than  major  system(s)  (excludes  ADP 
resources  as  defined  and  administered  under  AFR  300  series) 
(USAF/RD/LE  Policy  letter,  13  October  1981). 

Embedded  Computer  System  (ECS) 

(AF0TECP1) 

a)  A  computer  that  is  integral  to  an  electromechanical  system  and 
that  has  the  following  key  attributes: 

(1)  Physically  incorporated  into  a  large  system  whose  primary 
function  is  not  data  processing 

(2)  Integral  to,  or  supportive  of,  a  larger  system  from  a 
design,  procurement,  and  operations  viewpoint 

(3)  Inputs  include  target  data,  environmental  data,  command 
and  control,  etc. 

(4)  Outputs  include  target  information,  flight  information, 
control  signals,  etc. 

b)  In  general,  an  embedded  computer  system  (ECS)  is  developed, 
acquired,  and  operated  under  decentralized  management  (DoD 
Directives  5000.1,  5000.2). 

Emergency  MA 

(CURRENT) 

See  Maintenance  Priority. 

Engineering  Change  Proposal  (ECP) 

( AFR55-43) 

A  formal,  priced  document  (DD  Form  1692)  used  to  propose  changes 
to  the  contact  provisions  and  scope,  if  not  partially  waived  (see 
Contract  Change  Proposal),  and  to  the  configuration  item  baseline 
identification  especially  when  related  equipment,  critical  issues, 
interfaces,  or  technical  manuals  are  affected  or  retrofit  is 
involved.  See  MIL-STDs  480,  481,  and  483;  and  AFR  400-3. 

Enhancement  (Perfective)  MA 

(CURRENT) 

See  Maintenance  Type. 

Estimated  Person  Months  Per  Change 
(CURRENT) 

See  Person  Months  Per  Change 
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(CURRENT) 

See  Software  Supportabi 1 ity  Risk 
Estimation 
(ROWE) 

The  assignment  of  probability  measures  to  a  postulated  future 
event. 

Evaluated  Person  Months  Per  Change 
(CURRENT) 

See  Person  Months  Per  Change 
Evaluated  Risk 
(CURRENT) 

See  Software  Supportabi 1 ity  Risk. 

Evaluation 

(ROWE) 

Comparison  of  an  activity  performance  with  the  objectives  of  the 
activity  and  assignment  of  a  success  measure  to  that  performance. 

Evaluation  Criteria 

(AF0TECP1) 

Standards  by  which  achievement  of  required  operational  effective¬ 
ness/suitability  characteristics  or  resolution  of  technical  or 
operational  issues  may  be  judged.  For  full-scale  development  and 
beyond,  evaluation  criteria  must  include  quantitative  goals  (the 
desired  value)  and  thresholds  (the  value  beyond  which  the  charac¬ 
teristic  is  unsatisfactory)  whenever  possible  (DoD  Directive 
5000.3). 

Expandability 

(CURRENT) 

A  measure  of  the  extent  that  a  physical  change  to  information, 
computational  functions,  data  storage,  or  execution  time  can  be 
easily  accomplished  once  the  nature  of  what  is  to  be  changed  is 
understood. 


(AF0TECP5) 

A  measure  of  the  ease  with  which  the  func 
computer  hardware  or  software  may  be  expanded. 


the  functional  capability  of 
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Facility 

(AF0TECP5) 

The  physical  plant  and  the  services  it  provides;  specific  examples 
are  physical  space,  electrical  power,  physical  and  electromagnetic 
(TEMPEST)  security,  envi ronmental  control,  fire  safety  provisions, 
and  communications  availability. 

Firmware 

(AF0TECP1) 

a)  Computer  programs  and  data  loaded  in  a  class  of  memory  that 
cannot  be  dynamically  modified  by  the  computer  during 
processing. 

b)  Hardware  that  contains  a  computer  program  and  data  that  cannot 
be  changed  in  its  application  environment. 

Note  1.  Computer  programs  and  data  contained  in  firmware  are 
classified  as  software;  the  circuitry  containing  the  computer 
program  and  data  is  classified  as  hardware  (Data  and  Analysis 
Center  for  Software). 

Functional  Baseline 

(0oD480A) 

See  Basel ine. 

Functional  Conf iguration  Audit  (FCA) 

(0oD480A) 

The  formal  examination  of  functional  characteristics  test  data  for 
a  configuration  item,  prior  to  acceptance,  to  verify  that  the  item 
has  achieved  the  performance  specified  in  its  functional  or 
allocated  configuration  identification. 

Functional  Configuration  Identification 

(OoD480A) 

The  current  approved  technical  documentation  for  a  configuration 
item  which  prescribes  (1)  all  necessary  functional  characteris¬ 
tics,  (2)  the  tests  required  to  demonstrate  achievement  of 
specified  functional  characteristics,  (3)  the  necessary  interface 
characteristics  with  associated  Cl's,  (4)  the  Cl's  key  functional 
characteristics  and  its  key  lower  level  Cl's,  if  any,  and 
(5)  design  constraints,  such  as  envelope  dimensions,  component 
standardization,  use  of  inventory  items,  integrated  logistics 
support  pol icies . 
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High  Complexity  MA 
(CURRENT) 

See  Maintenance  Complexity. 

Historical  Maintenance  Profile 
(CURRENT) 

A  histogram  of  data  on  software  system  releases,  with  the  x-axis 
representing  discrete  ranges  of  (available)  person-months  per 
change  and  the  y-axis  representing  the  number  of  software  system 
releases  that  fall  into  each  x-axis  discrete  range.  For  purposes 
of  analysis  or  illustration,  the  axes  may  be  reversed. 

Independent  Verification  and  Validation  ( IV&V) 

(AF0TECP1) 

An  independent  assessment  process  structured  to  ensure  that 
computer  programs  fulfill  the  requirements  stated  in  system  and 
subsystem  specifications  and  satisfactorily  perform  the  functions 
required  to  meet  the  user's  and  supporter's  requirements .  IV&V 
consists  of  three  essential  elements:  independent,  verification, 
and  validation: 

(1)  Independent,  An  organization/agency  which  is  separate  from 
the  software  development  activity  from  a  contractual  and 
organizational  standpoint. 

(2)  Verification.  The  evaluation  to  determine  whether  the 
products  of  each  step  of  the  computer  program  development 
process  fulfill  all  requirements  levied  by  the  previous 
step. 

(3)  Validation.  The  integration,  testing,  and/or  evaluation 
activities  carried  out  at  the  system/subsystem  level  to 
evaluate  the  developed  computer  program  against  the  system 
specifications  and  the  user's  and  supporter's  requirements 
(AFR  88-14). 


Initial  Operational  Capability  (IOC) 


(CURRENT) 

That  point  in  a  system's  life  cycle  when  the  agreed  upon 
production  systems  has  been  delivered  to  the  user  (usin 
for  operational  use. 


upon  number  of 
using  command) 


Instrumentation 


(CURRENT) 

A  measure  of  the  extent  that  software  products  contain  aids  that 

enhance  testing. 
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Interface  Control  Working  Group  (ICWG) 

(MIL-STD-483) 

For  programs  which  encompass  a  system/HWCI/CSCI  design  cycle,  an 
ICWG  normally  is  established  to  control  interface  activity  between 
contractors  or  agencies,  including  resolution  of  interface 
problems  and  documentation  of  interface  agreements. 

Interoperabi 1 ity 

(AF0TECP5) 

A  measure  of  the  degree  to  which  computer  hardware/software  can 
interface  to  and  operate  with  other  similar  computer 
hardware/software. 

Low  Complexity  MA 

(CURRENT) 

See  Maintenance  Complexity. 

Maintainability 

(AF0TECP5) 

The  probability  that  a  system  out  of  service  for  maintenance  can 
be  properly  repaired  and  returned  to  service  in  a  stated  elapsed 
time. 

Maintenance  Complexity 


(CURRENT) 

The  general  degree  of  difficulty 
request:  high,  medium,  low. 


complete 


maintenance 


High:  An  MA  where  changes  are  in  requirements,  design,  code,  and 
test;  or  greater  than  10  percent  of  CSCI  is  affected;  or  several 
modules  are  affected  by  the  change  (global  changes);  or  the  tech¬ 
nical  nature  of  the  change  requires  highly  specialized  personnel 
skills;  or  the  level  of  effort  by  personnel  is  large. 

Medium:  An  MA  where  changes  are  in  design,  code  and  test;  or 
between  1  percent  and  10  percent  of  CSCI  is  affected;  or  at  least 
two  modules  are  affected  by  the  change  (semi-local);  or  the  level 
of  effort  by  personnel  is  average. 

Low:  An  MA  where  changes  are  isolated  to  only  one  unit  (e.g.,  one 
module/compilation  unit)  of  code;  or  no  more  than  1  percent  of 
CSCI  is  affected;  or  the  level  of  effort  by  personnel  is  minimal. 


31-13 


,  V  '  V.V.V/.W 


I 


K 


r  JM'  -<(  al(  , 


\  m'  I  |  IJ '  la  *-l  *mS  • 


THE  BOM  CORPORATION 


BDM/A8Q-86-0360-TR 


Maintenance  Documentation 


(AF0TECP5) 

The  documentation  that  describes  the  maintenance  of  computer 
system  hardware  and  software. 


Maintenance  Priority 


(CURRENT) 

The  criticality  of  the  maintenance  request  in  order  to  preserve 
mission  readiness;  emergency,  urgent,  normal. 


Emergency;  An  MA  requiring  all  available  personnel's  dedicated 
effort  to  correct  the  problem  as  soon  as  possible  (e.g., 
24  hours);  MIL -STD-1679  severity  code  1  or  2:  mission  termination 
or  severe  degradation. 


Urgent:  An  MA  requiring  next  "block  release"  turnaround; 

MIL-STD-1679  severity  code  3:  mission  impact. 


Normal:  An  MA  not  in  the  Emergency  or  Urgent  categories; 

MIL-STD-1679  severity  code  4  or  5:  mission  inconvenience. 


Maintenance  Profile 


(CURRENT) 

See  Historical  Maintenance  Profile. 


Maintenance  Request  Category 


(CURRENT) 

The  identification  of  a  maintenance  request  by  specification  of 
the  maintenance  priority,  type,  and  complexity. 


Maintenance  Type 


(CURRENT) 

The  type  of  maintenance  actions  required  to  complete  a  maintenance 
request:  conversion,  enhancement,  correction. 


Conversion  (Adaptive)  MA:  Any  change/effort  to  a  software  system 
which  is  initiated  as  a  result  of  changes  in  the  environment 
(e.g.,  hardware,  system  software)  in  which  the  software  system 
must  operate. 


Enhancement  (Perfective)  MA:  Any  change,  insertion,  deletion, 
modification,  or  extension  made  to  a  software  system  to  meet  the 
evolving  needs  of  the  user. 


Corrective  MA:  Any  change  which  is  necessitated  by  actual  faults 
(induced  or  residual)  in  a  software  system. 
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Medium  Complexity  MA 
(CURRENT) 

See  Maintenance  Complexity. 

Modularity 

(CURRENT) 

A  measure  of  the  extent  that  a  logical  partitioning  of  software 
products  into  parts,  components,  and/or  modules  has  occurred. 

Normal  MA 

(CURRENT) 

See  Maintenance  Priority. 

Operation  Support  Activity 
(CURRENT) 

Those  organizations  responsible  for  post  deployment  operation  and 
support  of  a  system.  Organizations  include  the  using  command, 
supporting  command,  contractors  (if  used),  and  test  and  evaluation 
agencies  (if  used) . 

Operational  Effectiveness 

(AF0TECP1) 

The  overall  degree  of  mission  accomplishment  of  a  system  used  by 
representati ve  personnel  in  the  context  of  the  organization, 
doctrine,  tactics,  threat  (including  countermeasures  and  nuclear 
threats),  and  environment  in  the  planned  operational  employment  of 
the  system  (DoD  Directive  5000.3). 

Operational  Suitability 

(AF0TECP1) 

The  degree  to  which  a  system  can  be  satisfactorily  placed  in  field 
use,  with  consideration  being  given  to  availability,  compatibil¬ 
ity,  transportability,  interoperabi 1 ity,  reliability,  wartime 
usage  rates,  maintainability,  safety,  human  factors,  manpower 
supportabi 1 ity,  logistic  supportabi 1 ity,  and  training  requirements 
(DoD  Directive  5000.3). 

Person-Months  per  Change  (PMPC) 


(CURRENT) 

Available  PMPC:  Raw  personnel  resources  workload  to  support  a 
user/supporter  baseline  workload  estimate  of  a  specified  number  of 
changes.  Computed  as  the  number  of  full-time  equivalent  personnel 
times  the  release  cycle  in  months  divided  by  the  total  number  of 
changes. 
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Estimated  PMPC:  An  estimate  of  a  personnel  resources  workload 
required  to  support  the  user/supporter  baseline  estimate.  This 
estimate  is  computed  by  using  a  regression  equation  whose 
coefficients  are  derived  from  historical  maintenance  release  data. 

Evaluated  PMPC:  A  realistic  estimate  of  personnel  resources  work¬ 
load  effectiveness  to  support  the  user/supporter  baseline  estimate 
as  derived  from  an  evaluation  of  the  software  supportabil ity  char- 
acteri sties . 

Personnel 

(CURRENT) 

See  Support  Personnel . 

Personnel  Skill  Level 
(CURRENT) 

A  subjective  integer  rating  from  1  (lowest)  to  5  (highest)  of 
software  support  personnel  experience,  education,  and  specific 
task  responsibility  capabilities. 

Physical  Configuration  Audit  (PCA) 

(DO0480A) 

The  formal  examination  of  the  "as-built"  configuration  of  a  unit 
of  a  Cl  against  its  technical  documentation  in  order  to  establish 
the  Cl's  initial  product  configuration  identification. 

Priority 

(CURRENT) 

See  Maintenance  Priority. 

Probability 

(ROWE) 

A  numerical  property  attached  to  an  activity  or  event  whereby  the 
likelihood  of  its  future  occurrence  is  expressed  or  clarified. 

Probability  Distribution 

(ROWE) 

The  representation  of  a  repeatable  stochastic  process  by  a 
function  satisfying  the  axioms  of  probability  theory. 

Probability  of  Occurrence 

(ROWE) 

The  probability  that  a  particular  event  will  occur,  or  will  occur 
in  a  given  interval . 
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Procurement  Activity 
(CURRENT) 

Those  government  organizations  responsible  for  assuring  delivery 
of  a  production  system.  Organizations  include  the  program  office, 
implementing  command,  development  and  operational  test  and  evalua¬ 
tion  agencies,  and  appropriate  independent  verification  and 
validation  agencies  if  used. 

Product  Baseline 

(DoD480A) 

See  Basel ine. 

Product  Configuration  Identification 
(0oD480A) 

The  current  approved  or  conditionally  approved  technical  documen¬ 
tation  which  defines  the  configuration  of  a  Cl  during  the 
production,  operation,  maintenance,  and  logistics  support  phases 
of  its  life  cycle,  and  which  prescribes  (1)  all  necessary  physical 
or  form,  fit  and  function  characteristics  of  a  Cl,  (2)  the 
selected  functional  characteristics  designated  for  production 
acceptance  testing,  and  (3)  the  production  acceptance  tests. 

Program  Management  Directive  (PMD) 

(AFR800-14) 

The  official  HQ  USAF  management  directive  used  to  provide 
direction  to  the  implementing  and  participating  commands  and 
satisfy  documentation  requirements .  It  will  be  used  during  the 
entire  acquisition  cycle  to  state  requirements  and  request  studies 
as  well  as  initiate,  approve,  change,  transition,  modify  or  ter¬ 
minate  programs.  The  content  of  the  PMD,  including  the  required 
HQ  USAF  review  and  approval  actions,  is  tailored  to  the  needs  of 
each  individual  program  (AFR  800-2). 

Program  Management  Plan  (PMP) 

(AFR800-14) 

The  document  developed  and  issued  by  the  Program  Manager  which 
shows  the  integrated  time-phased  tasks  and  resources  required  to 
complete  the  task  specified  in  the  PMD.  The  PMP  is  tailored  to 
the  needs  of  each  individual  program  (AFR  800-2). 

Program  Management  Responsibility  Transfer  (PMRT) 

(AFR800-14) 

That  point  in  time  when  the  designated  Supporting  Command  accepts 
program  management  responsibilities  from  the  Implementing  Corranand. 
This  includes  logistic  support  and  related  engineering  and  pro¬ 
curement  responsibilities  (AFR  800-4). 
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Program  Support  Tools 
(AF0TECP3) 

General  debug  aids,  test/retest  software,  trace  software/hardware 
features,  use  of  compiler/! ink  editor,  library  management/con¬ 
figuration  management/text  editor/display  software  tools. 

Program  Test  Plan 

(AF0TECP3) 

Set  of  descriptions  and  procedures  for  how  the  program  is  to  be 
(or  can  be,  or  has  been)  tested. 

Quality  Assurance  (QA) 

(CURRENT) 

All  actions  that  are  taken  to  assure  that  a  development  organiza¬ 
tion  delivers  products  that  meet  performance  requirements  and 
adhere  to  standards  and  procedures. 

Release 

(CURRENT) 

A  version  of  a  software  system  representing  either  the  initial 
baseline  configuration  or  an  update  to  a  previous  version  that 
incorporates  a  defined  set  of  software  change  requests.  Each 
release  becomes  a  new  baseline  configuration. 

Release  Engineering  Completion  Data 

(CURRENT) 

The  date  when  the  software  engineering  activity  for  a  release  is 
complete.  The  software  engineering  activity  includes  configura¬ 
tion  management,  quality  assurance,  and  software  maintenance 
project  phases  of  requirements,  design,  code,  unit  test,  integra¬ 
tion  test,  and  operational  test.  Activity  including  “kit 
proofing,"  prom  burning,  and  in  general  technical  order  modifica¬ 
tions  which  typically  occur  between  engineering  completion  and 
field  implementation  (distribution)  is  not  included. 

Release  Field  Date 

(CURRENT) 

The  date  when  a  software  system  release  is  officially  distributed 
and  implemented  in  the  field  for  operational  use. 

Release  ID 

(CURRENT) 

A  unique  identifier  for  a  software  system  release. 
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Release  Start  Date 
(CURRENT) 

The  date  when  major  analysis  activity  related  to  a  specified 
release  begins  for  which  software  support  resources  are  required. 

Rel  iabi 1 ity 

(ROWE) 

The  probability  that  the  system  will  perform  its  required 
functions  under  given  conditions  for  a  specified  operating  time. 

Risk 

(ROWE) 

The  potential  for  realization  of  unwanted,  negative  consequences 
of  an  event. 

Risk  Acceptance 

(ROWE) 

Willingness  of  an  individual,  group,  or  society  to  accept  a 
specific  level  of  risk  to  obtain  some  gain  or  benefit. 

Risk  Acceptance  Function 

(ROWE) 

A  subjective  operator  relating  the  levels  of  probability  of 
occurrence  and  value  of  a  consequence  to  a  level  of  risk 
acceptance. 

Risk  Acceptance  Level 

(ROWE) 

The  acceptable  probability  of  occurrence  of  a  specific  consequence 
value  to  a  given  risk  agent. 


Risk  Acceptance  Utility  Function 
(ROWE) 

The  profile  of  the  acceptability  of  the  probability  of  occurrence 
for  all  consequences  involved  in  a  risk  situation  for  a  specific 
risk  agent. 

Risk  Agent 

(ROWE) 

A  person  or  group  of  persons  who  evaluates  directly  the 
consequences  of  a  risk  to  which  the  person  or  group  of  persons  is 
subjected. 
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Risk  Assessment 
(ROWE) 

The  total  process  of  quantifying  a  risk  and  finding  an  acceptable 
level  of  that  risk  for  an  individual,  group,  or  society.  It 
involves  both  risk  determination  and  risk  evaluation. 

Risk  Assessment  Methodology  for  Software  Supportability  (RAMSS) 

(CURRENT) 

A  method  of  determining  the  disparity  between  the  estimated  risk 
(determined  from  the  support  concept,  baseline  software  support- 
ability  profile,  and  historical  maintenance  profile)  and  the 
evaluated  risk  (determined  from  a  conversion  of  the  software 
supportability  evaluation  metrics). 

Risk  Consequence 

(ROWE) 

The  impact  to  a  risk  agent  of  exposure  to  a  risky  event. 

Risk  Determination 
(ROWE) 

The  process  of  identifying  and  estimating  the  magnitude  of  risk. 
Risk  Estimation 
(ROWE) 

The  process  of  quantification  of  the  probaoi 1 ities  and  consequence 
values  for  an  identified  risk. 

Risk  Evaluation 

(ROWE) 

The  complex  process  of  developing  acceptable  levels  of  risk  to 
individuals  or  society. 

Risk  Profile  Basel ine 

(CURRENT) 

The  measure  of  information  and/or  requirements  which  serve  as  the 
zero  reference  against  which  negative  (and  positive)  outcomes  can 
be  determined. 

Risk  Reduction 

(ROWE) 

The  action  of  lowering  the  probability  of  occurrence  and/or  the 
value  of  a  risk  consequence,  thereby  reducing  the  magnitude  of  the 
risk. 
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Sensitivity  Analysis 


(ROWE) 

A  method  used  to  examine  the  operation  of  a  system  by  measuring 
the  deviation  of  its  nominal  behavior  due  to  perturbations  in  the 
performance  of  its  components  from  their  nominal  values. 


Simplicity 


(CURRENT) 

A  measure  of  the  extent  that  software  products  reflect  the  use  of 
singularity  concepts  and  fundamental  structures  in  organization, 
language,  and  implementation  techniques. 


Simulation 


(AFR800-14) 

The  representation  of  physical  systems  or  phenomena  by  computers, 
models  or  other  equipment. 


(CURRENT) 

A  software  support  site,  or  particular  location,  where  software 
support  activity  is  being  accomplished.  Includes  sites  such  as 
the  Air  Logistics  Centers  (ALCs). 


Site  Survey  Form 


(CURRENT) 

The  data  collection  form  used  during  the  software  support  site 
visits  to  collect  background,  evaluation,  and  maintenance  release 
data. 


Software 


(AF0TECP1) 

A  set  of  computer  programs,  procedures,  and  associated  documenta¬ 
tion  concerned  with  the  operation  of  a  data  processing  system. 


(CURRENT) 

The  programs  which  execute  in  a  computer.  The  data  input,  output, 
and  controls  upon  which  program  execution  depends  and  the  documen¬ 
tation  which  describes,  in  a  textual  medium,  development  and 
maintenance  of  the  program. 


Software  Change  Request 


(CURRENT) 

An  official  request  that  could  involve  a  change  to  a  software 
system.  Such  requests  include  problem  report,  enhancement 
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requirement,  modification  request,  or  any  other  form  that  is 
officially  tracked  by  a  configuration  management  function. 

Software  Configuration  Management 

(CURRENT) 

A  discipline  applying  technical  and  administrative  direction  and 
surveillance  to  1)  identify  and  document  the  functional  and 
physical  characteristics  of  a  configuration  item,  2)  control 
changes  to  those  characteristics,  and  3)  record  and  report  change 
processing  and  implementat ion  status. 

Software  Delivery 

(CURRENT) 

That  point  in  the  software  life  cycle  when  the  software  support 
function  assumes  responsibility  for  the  "next"  set  of  configura¬ 
tion  changes  to  the  software  (e.g.,  next  block  release).  This 
point  is  logically  no  later  than  PMRT,  but  could  be  as  early  as 
IOC.  This  applies  when  a  contractor  or  government  agency  assumes 
the  software  support  function. 

Software  Error 

(CURRENT) 

The  human  decision  (inadvertent  or  by  design)  which  results  in  the 
inclusion  of  a  fault  in  a  software  product. 

Software  Fault 

(CURRENT) 

The  presence  or  absence  of  that  part  of  a  software  product  which 
can  result  in  software  failure. 

Software  Life  Cycle  Process 

(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  in  a 
software  environment  to  the  software  development  and  support  life 
cycle  activities. 

Software  Maintainability 

( AF0TECP1 ) 

The  ease  with  which  software  can  be  changed  in  order  to: 

(1)  Correct  errors 

(2)  Add/modify  system  capabilities  through  software  changes 

(3)  Delete  features  from  programs 

(4)  Modify  software  to  be  compatible  with  hardware  changes. 
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(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to  per¬ 
form  software  maintenance  actions. 

Software  Maintenance 

(CURRENT) 

Those  actions  required  for: 

(1)  Correction  -  Removal,  correction  of  software  faults 

(2)  Enhancement  -  Addition/deletion  of  features  from  the 
software 

(3)  Conversion  -  Modification  of  the  software  because  of 
environment  (data  hardware)  changes. 

Software  Maintenance  Environment 

(CURRENT) 

An  integration  of  personnel  support  systems  and  physical 
facilities  for  the  purpose  of  maintaining  software  products. 

Software  Maintenance  Measures 

(CURRENT) 

Measures  of  software  maintainability  and  environment  capabilities 
to  support  software  maintenance  activity. 

Software  Maintenance  Project  Management 

(CURRENT) 

The  software  life  cycle  process  management  applied  during  the 
support  phase  for  the  software  to  accomplish  specific  software 
maintenance  tasks  which  derive  from  software  problem  reports  or 
change  requests. 

Software  Management 

(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  in  a 
software  environment  to  the  software  development/maintenance 
activities.  Also,  those  personnel  with  software  management 
responsibilities. 

Software  Project  Management 

(CURRENT) 

See  Software  Management. 

Software  Project  Management  Design  Methods 
(CURRENT) 

The  software  project  management  process  utilizes  design  methods 
which  enhance  software  supportabil ity  to  the  extent  that  design 
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methfiwlogy  standards  and  conventions  are:  1)  documented, 
followed,  and  validated  through  quality  assurance,  2)  can  be 
transitioned  to  support  activity,  and  3)  produce  adequate  design 
specifications  which  reflect  supportabi 1 ity  characteristics. 

Software  Project  Management  Implementation  Methods 
(CURRENT) 

The  software  project  management  process  utilizes  implementation 
methods  which  enhance  software  supportability  to  the  extent  that 
implementation/coding/testing  methodology,  standards,  and  conven¬ 
tions  are:  1)  documented,  followed,  and  validated  through  quality 
assurance,  2)  can  be  transitioned  to  the  support  activity,  and 
3)  produce  supportable  production  products. 

Software  Project  Management  Organization  Structure 
(CURRENT) 

The  software  project  management  process  organization  structure 
enhances  software  supportability  to  the  extent  that  the  physical 
structure,  functional  responsibilities,  external  interfaces  and 
assigned  personnel  provide  for  continuity  over  the  software  life 
cycle  phases,  and  have  proper  interfaces  with  organizations 
responsible  for  software  support. 

Software  Project  Management  Planning 
(CURRENT) 

The  software  project  management  process  utilizes  planning  which 
enhances  software  supportability  to  the  extent  that  plans  for  the 
development,  test,  product  transfer,  operation  and  support  exist, 
have  been  implemented,  have  been  appropriately  coordinated  across 
activities,  and  satisfy  contractual  and/or  regulation  require¬ 
ments. 

Software  Project  Management  Project  Interfaces 
(CURRENT) 

The  software  project  management  possesses  organization  interfaces 
which  enhance  software  supportabi 1 ity  to  the  extent  that  external 
project  organization  relationships  and  responsibilities  are: 
1)  defined,  2)  provide  a  valuable  functional  role,  and 
3)  contribute  to  systematic  cost  effective  procurement,  develop¬ 
ment,  operation  and  support  processes. 

Software  Project  Management  Test  Strategies 
(CURRENT) 

The  software  project  management  process  utilizes  test  strategies 
which  enhance  software  supportability  to  the  extent  that  the  test 
plans,  descriptions,  procedures,  and  results  have  been: 
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1)  documented,  2)  can  be  transitioned  to  the  support  activity,  and 
3)  provide  for  a  consistent  and  systematic  process  for  verifying 
and  validating  that  software  requirements  have  been  satisfied. 

Software  Reliability 

(CURRENT) 

A  quality  of  software  which  reflects  the  probability  of  failure 
free  operation  of  a  software  component  or  system  in  a  specified 
environment  for  a  specified  item. 

Software  Portabil ity  ' 

(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to 
transfer  the  software  from  one  environment  (hardware  and  system 
software)  to  another. 

Software  Support  Concept 

(CURRENT) 

The  estimated  support  personnel  resources,  level  of  dedication  and 
expertise  of  the  support  personnel,  and  the  duration  of  the  block 
release  cycle. 

Software  Support  Facility  (SSF) 

(AF0TECP5) 

The  facility  which  houses  and  provides  services  for  the  support 
systems  and  personnel  required  to  maintain  the  software  for  a 
specific  ECS. 

Software  Support  Personnel 
(CURRENT) 

See  Support  Personnel . 

Software  Support  Resources 


< 


(CURRENT) 

The  totality  of  personnel,  systems,  physical  facilities,  and 
calendar  time  that  are  used/consumed  during  a  software  support 
release  effort. 

Software  Supportabi  1 i ty 

(CURRENT) 

A  measure  of  the  adequacy  of  personnel,  resources,  and  procedures 
to  facilitate: 

(1)  Modifying  and  installing  software 

(2)  Establishing  an  operational  software  baseline 

(3)  Meeting  user  requirements. 
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Software  Supportabi 1 ity  Evaluation 
(CURRENT) 

An  evaluation  to  derive  a  measure  of  how  well  a  software  system 
can  be  supported.  (See  Software  Supportabi 1 ity. ) 

Software  Supportabi 1 ity  Evaluation  Metrics 

(CURRENT) 

The  closed-form  questionnaire  scores  for  each  software  support- 
ability  character istic  in  a  software  supportabi 1 ity  evaluation  as 
well  as  the  values  computed  by  cumulating  lower  level  scores. 

Software  Supportabi 1 i ty  Magnitude  of  Risk  Consequence 

(CURRENT) 

The  level  of  impact  to  a  software  user  or  supporter  as  a  result  of 
the  risk  level  of  a  software  supportability  negative  outcome. 

Software  Supportability  Negative  Outcome 

(CURRENT) 

Any  outcome  for  which  the  software  support  resources  are  not 
adequate  to  accomplish  required  software  support . 

Software  Supportability  Risk 

(CURRENT) 

The  probability  at  a  given  point  during  the  software  support  phase 
that  the  software  maintenance  activity  specified  by  a  baseline 
software  supportability  profile  cannot  be  accomplished  with  the 
available  software  support  resources. 

Estimated  Software  Supportability  Risk:  An  estimate  of  the  soft¬ 
ware  supportability  risk  determined  by  the  area  under  a  normal 
distribution  curve.  The  area  is  the  part  under  the  curve  greater 
than  the  subject  software's  available  person-months  per  change 
value  as  computed  from  the  software  support  concept  and  baseline 
software  change  profile.  The  normal  distribution  curve  is  deter¬ 
mined  by  using  the  estimated  person  months  per  change  as  the  mean 
and  the  standard  deviation  from  the  derivation  of  the  estimated 
person  months  per  change  regression  equation. 

Acceptable  Software  Supportability  Risk:  The  estimated  software 
supportability  risk  which  is  agreed  upon  by  the  user  (using 
command)  and  supporter  (supporting  command)  as  a  result  of  the 
baseline  software  supportability  agreement. 

Evaluated  Software  Supportability  Risk:  An  approximation  to  the 
software  supportability  risk  computed  from  the  software  support- 
ability  evaluation  metrics.  The  computation  is  derived  from  a 
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linear  regression  model  using  the  software  life  cycle  process, 
software  product,  support  personnel,  support  systems,  and  support 
facility  as  the  five  regression  equation  factors. 

Measured  Software  Supportabi 1 i ty  Risk:  See  Evaluated  Software 

Supportabil ity  Risk. 

Software  System 

(CURRENT) 

A  set  of  software  (specifications,  programs,  and  data)  which 
constitutes  a  well-defined  major  function  or  group  of  functions. 

Typical  systems  include  avionics  OFP,  ground  based  communications, 
missile  guidance,  simulation,  threat  generator,  ATE,  and  electro¬ 
nic  warfare. 

Software  System  Type 

(CURRENT) 

One  of  seven  classifications  of  a  software  system's  primary 
functional  mission:  ATO,  ATE,  C-E,  EW,  OFP,  SIM,  SUP. 

ATD:  Aircrew  Training  Device  or  Operational  Flight  Trainer  for 
training  and  support  of  an  operational  system,  usually  in  the  form 
of  a  mockup  simulator. 

ATE:  Automatic  Test  Equipment  software  to  support  the  testing  of 
hardware  units  under  test  (UUT),  create  and  maintain  the  environ¬ 
ment  where  the  test  software  may  be  used,  or  prepare/analyze/main¬ 
tain  test  software. 

C-E:  Communications-Electronics  software  for  command  ana  control, 

communications,  surveillance  and  warning,  air  traffic  control, 
intelligence,  and  other  related  functions. 

EW:  Electronic  Warfare  software  that  involves  the  use  of  electro¬ 
magnetic  energy  and  performs  functions  either  separate  or  integral 
to  a  larger  airborne  or  ground  system. 


OFP:  Operational  Flight  Program  sof tware/f irmware  that  is 

integral  to  an  onboard  aircraft  computer  system  including  naviga¬ 
tion,  flight  control,  fire  control,  weapon  delivery,  electronic 
engine  control,  and  heads-up  display. 

SIM:  Simulation  Software  not  included  as  part  of  the  ATD, 

including  simulation  models. 

SUP:  Support  Software  including  application  support  software  and 

system  support  software  not  included  in  any  other  category. 
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Specification  Change  Notice  (SCN) 

(CURRENT) 

The  SCN  is  used  to  distribute  approved  page  changes  to  authorized 
users  of  baseline  documents  who,  in  turn,  are  responsible  for 
posting  the  updates. 

Source  Code 

(CURRENT) 

The  form  of  the  program  code  in  its  source  language. 

Standards 

AF0TECP3) 

Procedures,  rules,  and  conventions  used  for  prescribing 
disciplined  program  design  and  implementation. 

Support  Concept 

(CURRENT) 

The  software  support  concept  usually  specified  as  part  of  the 
CRISP  and  OS/CMP.  Also  includes  that  part  of  the  support  concept 
necessary  to  establish  the  acceptable  risk  from  a  baseline  soft¬ 
ware  change  profile:  standard  release  duration,  number  of  support 
personnel,  average  skill  level,  percentage  of  personnel  dedicated 
to  releases,  support  facility,  etc. 

Support  Facility 

(CURRENT) 

The  physical  facility  resources  that  must  be  available  for  the 
software  support  resources  to  accomplish  a  specific  task(s). 

Support  Personnel 

(CURRENT) 

A  general  term  for  personnel  (military,  OoO  civilian,  or  DoD  con¬ 
tractor)  whose  skills  are  necessary  to  directly  support  mission 
critical  system  software  maintenance.  Includes  but  is  not  limited 
to  management,  technical,  non-technical  support,  and  contractor 
personnel . 

Support  System 

(AF0TECP5) 

Any  automated  system  used  to  change,  test,  or  manage  the  con¬ 
figuration  of  ECS  software  and  associated  documentation.  Includes 
but  is  not  limited  to  Host  Processor,  Software  Bench,  Laboratory- 
Integrated  Test  Facility,  Operational-Integrated  Test  Facility, 
and  Configuration  Management  System. 
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Support  System  Facility 
(AF0TECP5) 

The  facility  resources  that  must  be  available  for  the  software 
support  resources  to  accomplish  a  specific  task(s). 

System  Software 

( AF0TECP5) 

All  of  the  software  that  is  part  of  the  software  support  facility 
computer  system.  It  is  never  or  seldom  accessed  directly  by  soft¬ 
ware  support  facility  personnel;  it  controls  the  processing  of 
application  software.  It  includes  the  Operating  System,  Source 
Code  Editor,  Language  Translator,  Link  Editor/Loader, 
Librarian/File  Manager,  Oata  Base  Manager,  and  Automated  Software 
Development  Tool. 

Test  and  Evaluation  Master  Plan  (TEMP) 

(AFR55-43) 

An  overall  Test  and  Evaluation  (T&E)  plan  designed  to  identify  and 
integrate  the  effort  and  schedules  of  all  T&E  to  be  done  in  an 
acquisition  program. 

Threshold 

(ROWE) 

A  discontinuous  change  of  state  of  a  parameter  as  its  measure 
increases.  One  condition  exists  below  the  discontinuity,  and  a 
different  one  above  it. 

Time  to  Complete  Maintenance  Request  (TC) 

(CURRENT) 

The  calendar  time  from  receipt  of  the  maintenance  request  by  the 
support  control  group  until  the  request  has  been  accepted  as  part 
of  an  operational  system  software  configured  release.  (This  does 
not  mean  the  configuration  is  released  or  distributed,  and  this 
time  does  not  include  this  additional  delay,  if  any.) 

Type 

(CURRENT) 

See  Maintenance  Type. 

Uncertainty 

(ROWE) 

The  absence  of  information;  that  which  is  unknown. 
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Urgent  MA 
(CURRENT) 

See  Maintenance  Priority. 

Verification/Validation  (of  computer  programs) 

( AFR800-14) 

the  process  of  determining  that  the  computer  program  was  developed 
in  accordance  with  the  stated  specification  and  sati sfactori ly 
performs,  in  the  mission  environment,  the  function(s)  for  which  it 
was  designed. 


